Scale, Drivers, and Limits of Representational Alignment Across Large Language Models

Jialin Wu; Meenakshi Khosla

cs.CLOct 10, 2026

Scale, Drivers, and Limits of Representational Alignment Across Large Language Models

Authors: Jialin Wu, Meenakshi Khosla

Abstract

Comparing large language models can reveal shared representational structure and systematic differences between them. However, aggregate similarity measures can obscure how agreement varies across geometric scales and which linguistic properties support it. We introduce two complementary methods to resolve where models agree and what supports that agreement. Standard representational similarity analysis (RSA) summarizes similarity across all stimulus pairs, potentially obscuring differences in alignment between nearby and distant inputs. Neighborhood-Weighted RSA (NW-RSA) addresses this limitation by weighting stimulus pairs according to their proximity in representational space, placing alignment from local neighborhoods to global geometry on a common scale. Counterfactual RSA (CF-RSA) tests which linguistic properties support alignment by comparing intact representations in one model with representations under controlled linguistic perturbations in another. Across diverse language models, we find strong agreement in coarse representational geometry but substantial divergence in local neighborhoods. Alignment is observed both across broadly sampled natural text and within linguistic equivalence classes that preserve particular aspects of meaning or structure. Counterfactual analyses reveal selective sensitivity: changes to logical scope reduce alignment substantially more than the tested changes to hierarchical syntax, event roles, or discourse structure. Alignment also decreases when sentence structure or lexical meaning is disrupted in word lists and jabberwocky sentences, suggesting that the regularities of natural language support stronger convergence. These findings refine claims of representational universality by showing that strong overall similarity need not imply agreement in how models represent finer linguistic distinctions.

Figures & tables

Figure 1: The three axes of alignment studied here: geometric locality (NW-RSA), linguistic locality (equivalence classes), and linguistic structure and regularity (perturbations, counterfactual mismatches, and off-manifold inputs).

Figure 2: Two complementary RSA diagnostics (schematic; all values synthetic). Marker shapes identify the stimuli. Left: NW-RSA (Section 3.1.1 ). Each square holds two symmetric matrices: model

A

’s RDM or the binary

k

-NN mask (

k=2

) in the upper triangle, and model

B

’s RDM or the graded weights (

\tau=1

) in the lower triangle; neighborhood edges are colored by model. The shaded inset is a toy simulation with 20 points across 5 classes, where color marks the class rather than the model. In case 1, the two representations preserve the exact within-class geometry but rearrange the class centers, mimicking shared neighborhoods but a different coarse taxonomy: NW-RSA is

1.00

for

k\leq 3

, while global RSA is only

0.14

. In case 2, the class centers are shared but the instances are permuted within each class, mimicking the reverse: local NW-RSA is

-0.16

k=1

, while global RSA is

0.93

. Right: CF-RSA (Section 3.1.2 ). Color marks the model, and hollow markers are model

B

’s perturbed inputs

g(x_{i})

, placed in its intact geometry. The schematic RDM holds the distances from each

g(x_{i})

to the intact

x_{j}

(lines show row 1), excluding matched pairs; this is the transpose of

D^{B,g}

and yields the same correlation. Correlating

D^{A}

with

D^{B}

gives global RSA, and correlating

D^{A}

with

D^{B,g}

gives CF-RSA.

Figure 3: NW-RSA across geometric scales (mean over model pairs; shaded bands

\pm 1

SEM). Left: binary

k

-NN neighborhoods as a proportion

k_{\mathrm{prop}}

of stimuli. Right: graded temperature

\tau

. Solid lines show NW-RSA and dashed lines global RSA for DCLM, FineWeb, Control ( Tuckute et al., 2024 ) , and its baseline subset Control (T).

Figure 4: Mean RSA over model pairs (

\pm 1

SEM). Left: global RSA and RSA within linguistic equivalence classes (semantic, syntactic, entity, discourse, task). Right: intact sentences compared with meaning-preserving paraphrases, word-scrambled inputs, and word lists.

Axis	Model A input	Model B input
Hierarchical syntax ( Shivagunde et al., 2023 )	The journalist investigated which athlete the team had recruited.	The journalist investigated which team the athlete had joined.
Event roles ( McCoy et al., 2019 ; Zhang et al., 2019 )	The president advised the doctor.	The doctor advised the president.
Logical scope ( Marelli et al., 2014 )	Nobody is riding the bicycle on one wheel.	A person is riding the bicycle on one wheel.
Discourse structure ( Chen et al., 2019 ; Mostafazadeh et al., 2016 )	The boy kicked the ball. The ball rolled away.	The ball rolled away. The boy kicked the ball.

Table 1: Examples of mismatched perturbations used for Counterfactual RSA. The paired inputs share many surface properties but differ in interpretation, isolating structure from surface form.

Figure 5: Counterfactual (mismatch) alignment over four phenomena, pooled over model pairs (

\pm 1

SEM). Original (red) vs. meaning-altering mismatched (blue) pairing, under RSA, Procrustes, and linear predictivity (left to right). Logical scope collapses under mismatch in every metric. Discourse structure, event roles, and hierarchical syntax barely change.

Figure 6: Off-manifold alignment at the penultimate layer (mean over model pairs,

\pm 1

SEM). Noun2Jab, Verb2Jab, and Adj2Jab replace one noun, verb, or adjective with a pseudoword. Natural sentences align best, yet substantial agreement persists for nonword lists.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Source	Scale
Geometric scale (§ 5.1 )	DCLM ( Li et al., 2024 ) , FineWeb ( Penedo et al., 2024 ) , Control ( Tuckute et al., 2024 )	$10{,}000$ sents each, Control $2000$ (Control (T) $1000$ )
Linguistic scale (§ 5.2 )	Generated equivalence classes	$5$ classes $\times\,150$ groups (anchor $+\,50$ neighbors)
Counterfactual (§ 5.3 )	ROLE, HANS+PAWS, SICK, DiscoEval+ROCStories	$750$ pairs per phenomenon
Perturbation (§ 5.4 )	Generated paraphrase pairs	$750$ anchors $\times$ {paraphrase, scrambled, word list}
Off-manifold (§ 5.5 )	Derived from sentences	$750$ sents $\times\,7$ conditions

Appendix

Table 2: Stimulus source and scale for each experiment. The generated equivalence-class and paraphrase datasets and the off-manifold conditions will be released. The four counterfactual datasets and the web corpora are public (Table 6 ).

Figure 7: Construction of the stimuli we build ourselves. (a) Equivalence-class neighborhoods, built separately for each of the five equivalence types in three stages. (1) The generator (DeepSeek-V4-Flash) proposes 150 anchors, code checks their word count, and the auditor (DeepSeek-V4-Pro) checks each anchor; rejected anchors are regenerated, for up to five audit rounds. (2) The generator writes 50 neighbors for each anchor, and code checks the length, duplicates, and wording diversity of every set. (3) The auditor reviews the complete dataset, reading each anchor together with its 50 neighbors; the neighbors are audited at this stage. A group rejected by either check receives a new neighbor set (brown dashed paths); after four failed sets or five failed audit rounds, its anchor is resampled (blue path), at most three times. Finally, we randomly chose 10 examples per type and found that all of them satisfied the constraints. (b) Off-manifold conditions, built in three stages. (1) Candidate sentences from seven public datasets are normalized, filtered, and screened, and a rejected candidate is replaced from the pool. (2) Six conditions are derived from each of the 750 anchors by rule, with pseudowords from Wuggy and no LLM generation. (3) A rule-based audit re-derives every condition and checks it against its anchor; a condition that fails is derived again. The released dataset holds the 750 anchors in seven conditions. The source datasets are listed in Table 4 , and the filter criteria and derivation rules are given in the text.

Type	rule	LLM	resampled	rule	LLM	calls	sentences
	Anchors			Neighbor sets		Invalid	Discarded
Task	0	74	1	36	32	10	3,475
Semantic	0	78	4	85	90	29	8,832
Syntactic	92	67	39	229	88	45	16,048
Entity	43	105	0	8	17	22	1,398
Discourse	0	27	0	0	0	3	27

Appendix

Table 3: Rejections during equivalence-class construction. Rule and LLM count candidates rejected by the code checks and by the auditor, and resampled counts anchors replaced after their group kept failing. Anchor columns count sentences, and neighbor columns count sets of 50 neighbors. Invalid calls returned malformed or incomplete neighbor sets and are not counted in the last column, which gives the discarded sentences recorded in the logs and is a lower bound.

Source	Subset	Sentences
SICK	contradiction subset	242
ROCStories	training set	213
PAWS	label-0 subset	132
STS-B	filtered	69
ROLE-1500	deduplicated	41
DCLM	web corpus	30

Appendix

Table 4: Human-written sources of the 750 off-manifold anchor sentences. Licenses and citations for each source are listed in Table 6 .

Condition	Meaning	Structure
Sentences	Yes	Yes
Word lists	Yes	No
Jabberwocky	No	Yes
Nonword lists	No	No
Noun $\rightarrow$ pseudoword	Partial	Yes
Verb $\rightarrow$ pseudoword	Partial	Yes

Appendix

Table 5: Off-manifold stimulus families. The

2\times 2

base removes word-level meaning and sentence structure in separate steps. Three graded conditions replace one content word with a pseudoword while retaining structure and most meaning.

Figure 8: Off-manifold alignment on the human-authored stimuli of Regev et al. (2024) (mean over model pairs,

\pm 1

SEM), read at the penultimate layer. Left: RSA; middle: Procrustes; right: linear predictivity.

Figure 9: Graded NW-RSA with the two models’ weights combined by their maximum instead of their geometric mean (mean over model pairs; shaded bands

\pm 1

SEM). Dashed lines mark global RSA.

Figure 10: Linguistic-scale analysis under mapping-based metrics (mean over model pairs,

\pm 1

SEM). Left: Procrustes; right: linear predictivity.

Figure 11: Paraphrase and word-scrambling analysis under mapping-based metrics (mean over model pairs,

\pm 1

SEM). Left: Procrustes; right: linear predictivity.

Figure 12: Off-manifold alignment under mapping-based metrics (mean over model pairs,

\pm 1

SEM). Top: Procrustes; bottom: linear predictivity.

Panel of Figure 13 — Figure 13: Layer-wise linguistic-scale analysis by relative network depth. Top left: RSA; top right: Procrustes; bottom: linear predictivity. Solid lines are the equivalence classes and the dashed line is the global sample. The syntactic class is the lowest at every depth, and the spread across classes narrows with depth.

Panel of Figure 14 — Figure 14: Layer-wise counterfactual contrast by relative network depth. Top left: RSA; top right: Procrustes; bottom: linear predictivity. Solid lines are the original pairing, dashed the mismatched pairing. The logical-scope gap is the largest at every depth and widens through the middle layers, while the discourse-structure, event-role, and hierarchical-syntax gaps remain small.

Panel of Figure 15 — Figure 15: Layer-wise paraphrase and word-scrambling analysis by relative network depth. Top left: RSA; top right: Procrustes; bottom: linear predictivity. Scrambled inputs align nearly as well as intact sentences at the first layer and decline with depth, while paraphrases move closer to intact sentences.

Panel of Figure 16 — Figure 16: Layer-wise off-manifold alignment by relative network depth. Top left: RSA; top right: Procrustes; bottom: linear predictivity. Natural sentences align best at every depth, and the gap to word lists, jabberwocky, and nonword lists is larger in later layers than in the first layer.

Panel of Figure 17 — Figure 17: Layer-wise off-manifold alignment on the human-authored stimuli of Regev et al. (2024) by relative network depth. Top left: RSA; top right: Procrustes; bottom: linear predictivity. In the earliest layers, nonword lists align as well as or better than sentences; beyond them, sentences align best.

Category	Item	License / Terms
Models	BERT, ALBERT, ELECTRA ( Devlin et al., 2019 ; Lan et al., 2020 ; Clark et al., 2020 )	Apache-2.0
	RoBERTa, MiniLM, DeBERTaV3 ( Liu et al., 2019 ; Wang et al., 2020 ; He et al., 2023 )	MIT
	GPT-2 ( Radford et al., 2019 )	MIT
	Mistral-7B ( Jiang et al., 2023 )	Apache-2.0
	Qwen-2.5 ( Yang et al., 2024 )	Apache-2.0 (3B: Qwen Research License)
	Llama-3.2 ( Dubey et al., 2024 )	Llama 3.2 Community License

Appendix

Table 6: Main existing assets used in this work and their licenses or usage terms. Exact Hugging Face identifiers and versions will accompany the code released upon acceptance.

Representation Alignment

Representation alignment studies how learned feature spaces can be compared or mapped to transfer meaning, capabilities, and information across models and tasks.

See all 111 papers

May 22, 2026

Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoning

Large language models trained under diverse objectives and architectures have been shown to develop increasingly similar internal representations, an observation formalized as the Platonic Representation Hypothesis. Whether this representational convergence extends to the reasoning processes that operate over shared representations remains untested. We evaluate representational similarity across 16 language models from 8 families (1.5B to 72B parameters) on 800 reasoning problems spanning mathematics, science, commonsense, and truthfulness, stratifying by problem difficulty, computational stage, and causal relevance. Our analysis reveals three dissociations: a difficulty inversion, where models converge more on problems they collectively fail (Centered Kernel Alignment [CKA] = 0.897) than on those they solve (CKA = 0.830); a generation gap, where pre-decision representations align (CKA = 0.875) while post-decision representations diverge (CKA = 0.274); and epiphenomenal correctness, where shared information is decodable across models (66% transfer accuracy) but exerts minimal causal influence on predictions (1.5% to 5.5% flip rate across ablation protocols). These results indicate that representational convergence in language models reflects shared input processing constraints rather than shared reasoning strategies, with direct implications for ensemble design, interpretability transfer, and evaluations of model similarity. Code is available at https://github.com/Usama1002/convergence-without-understanding.

Muhammad Usama, Dong Eui Chang

Control Laboratory, School of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Daejeon 34141, Republic of Korea · Control Laboratory, School of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST)2026 Daejeon 34141, Republic of Korea

Representational Similarity Analysis LLM Interpretability Reasoning Consistency in Language Models Representation Geometry in Language Models Platonic Representation Hypothesis LLM Reasoning

Ask PDF

Jan 14, 2026

Representational Similarity Alignment of Figurative Language in Humans and Large Language Models

Large language models (LLMs) often produce judgments similar to those of humans, but their interpretation of figurative language remains unclear. We compared human participants and four instruction-tuned LLMs (GPT-4, Gemma-2-9B, Llama-3.2, and Mistral-7B) on their ratings of 240 dialogue-based sentences spanning six linguistic traits: conventionality, sarcasm, funny, emotion, idiomacy, and Gen Z slang. While humans and LLMs were strongly aligned at the surface level, differences emerged in representational similarity patterns, particularly in idiomacy and Gen Z slang. GPT-4 showed the closest alignment to humans, though all models differed at the representational level, suggesting that similar outputs may arise from different underlying language representations. We also examined the relationship between model scale and representational similarity using the Gemma model series (Gemma-2-9B-IT, Gemma-3-27B-IT, and Gemma-4-31B-IT), and found that representational similarity patterns were largely unaffected by model size.

Samhita Bollepally, Aurora Sloman-Moll, Takashi Yamauchi +1

Figurative Language Understanding LLM Evaluation Pragmatic Reasoning in Language Models

Ask PDF

Oct 10, 2026

Do LLMs differentiate linguistic traits internally?: A layer-wise analysis

Large language models often produce judgments that resemble human responses, but it remains unclear whether they internally capture linguistic nuances such as conventional versus idiomatic meaning. A layer-wise representational analysis was conducted on two equal-sized pre-trained models, Llama-3.2-1B and Gemma-3-1B-PT, to examine whether and where these models differentiate linguistic traits from input. The analysis combined layer-wise probing with geometric distance measures, using the top embedding dimensions ranked by principal component loadings for each layer. The results show that even with equal capacity, models organize linguistic information in fundamentally different ways, suggesting the need to analyze internal representational geometry, examining not just behavioral outputs but semantic space alignment when evaluating how closely human and model language interpretation correspond.

Samhita Bollepally, Takashi Yamauchi, Manju Muralee +1

LLM Interpretability Representation Geometry in Language Models Language Model Probing

Ask PDF

Abstract

Representation Alignment

Explore similar work

Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoning

Representational Similarity Alignment of Figurative Language in Humans and Large Language Models

Do LLMs differentiate linguistic traits internally?: A layer-wise analysis