Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy

Juan Pablo Vigneaux; Mary Kennedy; Khalil Iskarous; Robert Frank; Matilde Marcolli

cs.CLAug 28, 2026

Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy

Authors: Juan Pablo Vigneaux, Mary Kennedy, Khalil Iskarous, Robert Frank, Matilde Marcolli

Organizations: Northwestern University · University of Southern California · Yale University · California Institute of Technology

Abstract

Structural probes were introduced by Hewitt and Manning to reconstruct syntactic trees from a neural language model's latent representations. They are evaluated by calculating the proportion of syntactic tree edges correctly reconstructed over an annotated corpus (as measured by undirected unlabeled attachment score). Here, we disaggregate this measure, considering undirected attachment score by label (UASL), which assesses the reconstruction accuracy of each syntactic relation separately, establishing important differences among relations that overlap linguistic distinctions. Moreover, we identify two factors that predict most of UASL's variability across relations: (i) the mean and dispersion of the linear distance (on a log scale) between the related words, and (ii) the diversity (similarity-aware entropy) of the syntactic relation's head. These results, which hold across a range of model sizes and architectures, shed light on the degree of abstraction of the representation of syntax in language models and the dependence of such representation on geometric properties of the embedding space.

Figures & tables

Figure 1: Held-out UASL against the similarity-corrected entropy of the relation’s head $H_{\rm head}^{\rm sim}$ , for the optimal structural probe trained on BERT-base embeddings of the PTB. Both axes are residuals with respect to the other two predictors in the weighted linear regression of § 6 . Each point is one relation, with area proportional to the number of edges used to measure its UASL; the ten largest are labelled.

Figure 2: UASL curves for verbs’ external and internal arguments. Edge counts per relation and split are in Table A.2 .

Figure 3: UASL curves of dependency relation that represent top, intermediate, and low performance, with intermediate performance further divided among high, middle and low range. Edge counts per relation and split are in Table A.2 .

Figure 4: Mean UASL against linear distance, for each model at its own optimal residual stream checkpoint, given in parentheses. The average is taken over the dependency relations with at least five gold edges at that distance, and the same decay appears in all five models: steep over the first few words, then close to flat. Figure C.1 gives the BERT-base curve at every post-block checkpoint.

Figure 5: Fitness of the log-linear decay model across selected dependency relations, with the residual stream evaluated after each full transformer block. Relations taken from Figures 2 and 3 , and § 5.1 ; csubj , iobj and mwe omitted because they vary too little in linear distance. Relations are ordered by mean $R^{2}$ across residual stream checkpoints. Auxpass is fitted on only three points. See Figure C.4 in the Appendix for elaboration.

Figure 6: Dendrogram of dependency relations, clustered by UASL performance as a function of linear distance, with symmetric log-scale on the horizontal axis (linear below 0.3, logarithmic above) and a vertical line marking 25% of the tallest merge.

Predictor	Coef	SE	p
const	1.045	0.038	$<$ .001
head sim-entropy	-0.082	0.014	$<$ .001
mean log linear dist.	-0.252	0.030	$<$ .001
sd log dep length	+0.163	0.066	.019

Table 1: WLS regression predicting UASL from properties of dependency relations, all computed on our dev set (from the PTB). Each relation is weighted by the number of dev set edges its UASL was computed from. Predictors are on their natural scales (

R^{2}=0.736

, adjusted

R^{2}=0.715

F(3,38)=35.2

p<10^{-10}

n=42

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Type	Layer norm.	Attn./MLP	Position encoding	$L$	$d$	ckpts	best
BERT-base	encoder	post-LN	sequential	learned absolute, in the stream	12	768	26	16
DeBERTa-v3-base	encoder	post-LN	sequential	disentangled relative, in attention	12	768	13	9
ModernBERT-base	encoder	pre-LN	sequential	rotary, in attention	22	768	46	30
GPT-2-base	decoder	pre-LN	sequential	learned absolute, in the stream	12	768	26	16
GPT-J-6B	decoder	pre-LN	parallel	rotary, in attention	28	4096	29	8
RoBERTa-Shuffle-n1	encoder	post-LN	sequential	learned absolute, in the stream	12	768	13	9

Appendix

Table A.1: The six models probed in this work. Type specifies the attention pattern and pre-training objective, Attn./MLP whether a block applies attention and MLP in sequence or in parallel, and Position encoding where positional information enters the computation;

L

is the number of blocks,

d

the model dimension, ckpts the number of residual stream checkpoints we introduced and best the one at which development UUAS peaks. In general, ckpts follows from the architecture, with the exception of DeBERTa-v3-base and RoBERTa-Shuffle-n1 for which we only extracted uncontextualized embeddings with positional information and post-transformer-block checkpoints.

Relation	Train	Dev	Test	Relation	Train	Dev	Test
acomp	1,163	48	99	neg	5,370	199	371
advcl	6,210	293	372	nn	74,082	3,228	4,364
advmod	29,282	1,276	1,839	npadvmod	3,451	189	231
amod	60,276	2,506	3,481	nsubj	66,179	2,780	3,991
appos	6,926	276	466	nsubjpass	6,625	253	386
aux	33,875	1,240	2,148	num	22,606	1,148	1,238

Appendix

Table A.2: Gold edge counts per dependency relation, for each Penn Treebank WSJ split. A relation’s count is the number of edges the probe evaluation scores under it: edges of the gold dependency tree whose two endpoints are both non-punctuation tokens. Root arcs are excluded, having only one endpoint among the tokens, as are the edges the evaluation inserts to bridge removed punctuation, which carry no single relation. The dev counts are the weights of the weighted regression of § 6 , and the test counts the denominators of the held-out UASL reported there. csubjpass occurs once in dev and not at all in test, so it drops out of every test-set analysis.

Panel of Figure 10 — Figure B.1: UASL by relation at residual stream checkpoints in different models. Clockwise from the top left: DeBERTa-v3-base, ModernBERT-base, GPT-J-6B, GPT-2-base. Compare Figure 2 .

Figure C.1: Average UASL and standard deviation of per-relation averages, as a function of linear distance between endpoints of a dependency edge, at all odd residual stream checkpoints (after each attention block) and at residual stream checkpoint 16 with optimal UUAS.

Panel of Figure 12 — Figure C.2: Mean UASL shown as a function of the linear distance for different models. Shaded bands represent one standard deviation across relations. Clockwise from the top left: DeBERTa-v3-base, ModernBERT-base, GPT-J-6B, GPT-2-base. In each case the upper part of the figure shows every checkpoint, while the lower part focused on the best one.

Figure C.3: Fitness of a log-linear decay model across dependency relations for BERT-base, with residual stream evaluated after each full transformer block, and at residual stream checkpoint 16. The colour scale runs from red ( $R^{2}=0$ ) through yellow to green ( $R^{2}=1$ ). Omitted are relations for which a linear regression could not be fit.

Figure C.4: Individual dependency relation data by log linear distance, underlying Figure 5 . Datapoints are UASL values for one linear distance $\delta$ with at least five gold edges in the test set. Probing is done at residual stream checkpoint 16, where BERT-base attains its peak UUAS and which was used to fit the regression of § 6 . The line is the least-squares fit of Equation 1 . A low $R^{2}$ need not indicate noise: ccomp spans the widest range of linear distances and is close to flat, so its near-zero $R^{2}$ reflects the absence of decay rather than scatter about one. auxpass , at the head of the ordering, is fitted on three points.

Panel of Figure 15 — Figure C.5: Goodness of fit of the log-linear decay model, for DeBERTa-v3-base (Left) and GPT-2-base (Right), both models involving twelve transformer blocks. In both cases a column is the residual stream after an entire transformer block; the two differ in checkpoint indices only because GPT-2 is stored with a post-attention checkpoint interleaved between consecutive post-MLP ones. Rows are the relations, ordered by mean $R^{2}$ across residual stream checkpoints.

Panel of Figure 16 — Figure C.6: Goodness of fit for the log-linear decay model in the case of ModernBERT-base (top) and GPT-J-6B (bottom), respectively with columns with residual stream checkpoints $3,5,\ldots,45$ , after each of the 22 blocks, and residual stream checkpoints $1$ – $28$ , after each of 28 blocks. Rows are as in Figure C.5 .

Relation	Value	Relation	Value
advcl	17	expl	4
ccomp	17	npadvmod	4
parataxis	17	pcomp	4
conj	13	pobj	4
nsubjpass	12	poss	4
cc	11	predet	4

Appendix

Table C.1: For each dependency relation, the table displays the value of

\delta_{90}(r)

: the 90th percentile of its linear distances over the corpus, the linear distance being the number of words separating the head and the dependent (so that

1

denotes adjacent words).

Figure C.7: The four average-linkage dendrograms of the 41 dependency relations. $\tilde{d}_{\mathrm{range}}$ measures the differences between the relations’ count-weighted 90th-percentile linear distance while $\tilde{d}_{\mathrm{UASL}}$ measures the probe’s UASL curves as a function of linear distance. Dendrograms are computed with a variation of no arc-function UASL ( $\alpha=0$ , top left, range-only) to only UASL performance as a function of linear distance ( $\alpha=1$ , bottom right, UASL only). The horizontal axis uses a symmetric-log scale (linear below 0.3, logarithmic above), and the dashed gray vertical line marks 25% of the tallest merge, after which leaf labels are colored to indicate their cluster affiliation.

Figure C.8: Like Figure C.7 , the above dendrograms of the 41 dependency relations measure the differences between the relations’ count-weighted 90th-percentile linear distance. However, unlike Figure C.7 , the range component has been replaced with the Wasserstein-1 distance described in Equation 14 . All other components are the same as in Figure C.7 : $\alpha=1$ (bottom right) is a dendrogram computed only with respect to UASL, while $\alpha=0$ (top left) is computed only with the new Wasserstein-1 range component, and the horizontal axis and gray vertical line correspond to that in Figure C.7 .

Panel of Figure 20 — Figure C.9: Model comparison: DeBERTa-v3-base, ModernBERT-base, GPT-J-6B, GPT-2-base (clockwise from top-left). The average-linkage dendrograms of the 41 dependency relations, clustered by UASL, as a function of linear distance ( $\alpha=1$ in the composite distance of Figure C.7 ), measured over all residual stream checkpoints of the test set. The horizontal axis is a symmetric-log scale, linear below 0.3 and logarithmic above; the dashed grey vertical line marks 25% of the tallest merge, past which leaf labels are coloured by cluster. The four panels share these settings, so they are stated here once rather than repeated as a title above each.

Model	Peak UUAS	ckpt	$R^{2}$	head sim-entropy	mean log length	sd log length
BERT-base	0.815	16	0.736	$-0.082^{***}$	$-0.252^{***}$	$+0.163^{*}$
DeBERTa-v3-base	0.788	9	0.609	$-0.067^{***}$	$-0.226^{***}$	$+0.186^{*}$
ModernBERT-base	0.762	30	0.700	$-0.056^{***}$	$-0.216^{***}$	$+0.103^{\phantom{*}}$
GPT-2-base	0.776	16	0.744	$-0.076^{***}$	$-0.218^{***}$	$+0.143^{*}$
GPT-J-6B	0.783	8	0.746	$-0.069^{***}$	$-0.193^{***}$	$+0.125^{*}$

Appendix

Table D.1: Peak UUAS and WLS regression of § 6 , evaluated for each model at its own optimal residual stream checkpoint. The symbol ∗∗∗ denotes

p<.001

, while ∗ denotes

p<.05

. The dispersion term is marginal for ModernBERT-base (

p=.095

) and significant in all the other cases. All three effects discussed in the main text occur in each model with the same sign and with comparable magnitude.

Figure D.1: Added-variable plots of UASL against each regression predictor, one row per model. Point area is proportional to the relation’s weight in the regression.

added term, $p$ -value
Model	$M_{1}$	$M_{2}$	$M_{3}$
BERT-base	0.678	0.715	0.719
DeBERTa-v3-base	0.523	0.579	0.605
ModernBERT-base	0.660	0.676	0.668
GPT-2-base	0.686	0.723	0.724
GPT-J-6B	0.691	0.726	0.719

Appendix

Table D.2: Adjusted

R^{2}

for the three moments, and

p

-value of the term added at each step (nested

F

-test): adding

M_{2}

helps while adding

M_{3}

does not.

Figure E.1: Dev set UASL curves of rank-64 structural probes trained on RoBERTa-Shuffle-n1’s latent representations. Relations are those of Figure 3 . Residual stream checkpoints numbered as in Hewitt and Manning (2019) .

Figure E.2: Mean UASL as a function of linear distance for RoBERTa-Shuffle-n1, over all 12 post-block residual stream checkpoints (upper panel) and at the best one alone (lower panel). The shaded bands represent one standard deviation across relations. The curve is nearly flat: linear distance carries almost no information.

Figure E.3: Goodness of fit for the log-linear decay model for RoBERTa-Shuffle-n1. Columns are residual stream checkpoints $1$ – $12$ , after each transformer block and rows are syntactic relations ordered by mean $R^{2}$ . The median $R^{2}$ is $0.102$ , against $0.777$ for BERT-base in Figure C.3 .

Predictor	Coef	SE	p	BERT
const	0.100	0.018	$<$ .001	1.045
head sim-entropy	$+0.014$	0.006	.031	$-0.082$
mean log dep length	$-0.002$	0.014	.870	$-0.252$
sd log dep length	$-0.060$	0.031	.058	$+0.163$

Appendix

Table E.1: The regression of Table 1 on RoBERTa-Shuffle-n1, evaluated at its best residual stream checkpoint (

R^{2}=0.195

, adjusted

R^{2}=0.132

F(3,38)=3.07

p=.039

n=42

). The final column repeats the BERT-base coefficient for comparison. The diversity and the dispersion coefficients change sign, and the linear distance coefficient is reduced to less than

1\%

of its magnitude.

Language Model Probing

Language model probing measures what information and capabilities models encode and how those signals relate to their behavior.

See all 265 papers

Sep 24, 2026

Encoded but Not Decoded: Layer-Localized Evidence for a Three-Level Gap in LLM Syntax

A language model can fail a syntactic test in two distinct ways: by not encoding the relevant structure, or by encoding it but failing to use it at the output. Behavioral evaluation alone cannot tell these apart. We propose a three-level evaluation framework (behavioral deployment, LM-head readout, and probe recoverability) measured on the same items under the same binary decision. Using a compact trilingual (English, Chinese, German) control-dependency benchmark, we find that probe recoverability exceeds or equals LM-head readout, which in turn exceeds or equals behavioral deployment, across seven models and all three languages in the aggregate. The recoverability surplus is never negative across all 14 (model, task) conditions. The disconnect concentrates in subject-control, where a nearest-noun heuristic gives the wrong answer. The single largest gap (0.653) appears on Qwen3-0.6B Instruct in question answering. The gap persists at Qwen3-14B Instruct. Instruction tuning degrades deployment more than encoding in percentage terms. We rule out option-position bias, late-layer erasure, output-formatting artifacts, and probe-training variance. The pattern is consistent with decoding that favors surface shortcuts, and the behavior-probe gap measures the strength of that preference. Activation patching shows the gap is layer-localized. Under instruction tuning, the LM-head-decoded layer shifts approximately ten layers later than the probe-decoded layer. These findings argue that behavioral evaluation understates what models encode, while probing alone overstates what they deploy.

Zhenyan Lu, He Wang, Xiaohui Huang

College of International Studies, National University of Defense Technology, Nanjing, China

LLM Evaluation Language Model Decoding Language Modeling Language Model Probing

Ask PDF

May 26, 2026

Probing Minimalist Phase Structure in LLMs: What Universal Dependencies Cannot Represent

Structural probes train on Universal Dependencies (UD), which does not encode formal-syntactic abstractions such as phase boundaries or phase-internal cohesion. Whether large language models (LLMs) encode these remains an open question that UD-based probing cannot answer by construction. We evaluate structural probes on wh-movement stimuli where UD distances are invariant across conditions by design -- any non-zero effect therefore reflects structure beyond UD. The three conditions -- bare small clause, infinitival, and finite -- are ordered by the number of Minimalist Program (MP) phase boundaries the wh-element crosses. Across 13 LLMs from four families, we find a phase-count gradient on a cross-clause pair (12/13 models) and a 13/13 sign asymmetry on a within-clause pair whose UD distance is identical across conditions -- the latter specifically predicted by phase-internal cohesion, an MP abstraction invisible to UD by construction. Activation patching confirms the representations are causally active in 12/13 models. These findings suggest that distributional pretraining can induce representations aligned with formal-syntactic abstractions beyond the reach of annotation-based probing; UD-grounded probes provide a lower bound on syntactic encoding, not an upper bound.

Yuanhao Chen, Peter Chin

Dartmouth College

Language Model Probing

Ask PDF

May 6, 2026

Implicit Representations of Grammaticality in Language Models

Grammaticality and likelihood are distinct notions in human language. Pretrained language models (LMs), which are probabilistic models of language fitted to maximize corpus likelihood, generate grammatically well-formed text and discriminate well between grammatical and ungrammatical sentences in tightly controlled minimal pairs. However, their string probabilities do not sharply discriminate between grammatical and ungrammatical sentences overall. But do LMs implicitly acquire a grammaticality distinction distinct from string probability? We explore this question through studying internal representations of LMs, by training a linear probe on a dataset of grammatical and (synthetic) ungrammatical sentences obtained by applying perturbations to a naturalistic text corpus. We find that this simple grammaticality probe generalizes to human-curated grammaticality judgment benchmarks and outperforms LM probability-based grammaticality judgments. When applied to semantic plausibility benchmarks, in which both members of a minimal pair are grammatical and differ in only plausibility, the probe however performs worse than string probability. The English-trained probe also exhibits nontrivial cross-lingual generalization, outperforming string probabilities on grammaticality benchmarks in numerous other languages. Additionally, probe scores correlate only weakly with string probabilities. These results collectively suggest that LMs acquire to some extent an implicit grammaticality distinction within their hidden layers.

Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu +2

Massachusetts Institute of Technology

Language Model Probing

Ask PDF

Abstract

Language Model Probing

Explore similar work

Encoded but Not Decoded: Layer-Localized Evidence for a Three-Level Gap in LLM Syntax

Probing Minimalist Phase Structure in LLMs: What Universal Dependencies Cannot Represent

Implicit Representations of Grammaticality in Language Models