Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Models

Tido Specht; Elias Benedict Krey; Nils Neukirch; Nils Strodthoff

cs.LGOct 1, 2026

Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Models

Authors: Tido Specht, Elias Benedict Krey, Nils Neukirch, Nils Strodthoff

Organizations: University of Groningen · Carl von Ossietzky University of Oldenburg

Abstract

Understanding information processing in large language models (LLMs) requires dissecting the geometric organization of their internal token representations. While existing mechanistic interpretability (MI) methods seek to extract concepts, they are constrained by a strong linearity assumption challenged by evidence of non-linear feature manifolds. We move beyond linear concepts by adapting Non-Linear Multi-Dimensional Concept Discovery (NLMCD) from computer vision to token-level LLM activations, modeling concepts as low-dimensional manifolds. To compare concept manifolds across layers and models, we introduce a concept-based alignment (CBA) score, a generalized Rand index that measures geometric proximity without explicit feature matching. Our analysis yields six key findings: (i) a neighboring-layer sanity check shows CBA is more sensitive than PCA- or CKA-based linear baselines; (ii) layer-by-layer alignment matrices reveal two block structures in intermediate and late layers, consistent across models and obscured by linear metrics; (iii) concept composition remains syntax-dominated through most of the network before giving way to increasingly mixed syntactic-semantic concepts in later layers, with increasing output-orientation toward the final layers; (iv) multilingual concept sharing between English and Mandarin is training-dependent rather than universal, strongest in Qwen, weaker in Llama, and absent in GPT-2; (v) inter-model alignment mirrors this structure, with strong correspondence between same-family Qwen models of different scale but weak alignment across model families; and (vi) across Tulu-3 training stages, alignment is highest between adjacent stages, with the largest shift between the base model and SFT, while subsequent preference-alignment stages (DPO, RLVR) leave early layers largely unchanged and RLVR mostly preserves DPO's concepts in late layers.

Figures & tables

Figure 1: Overview of NLMCD-NLP. Layer-wise token activations are clustered into nonlinear concepts with fuzzy memberships. Using a concept-based alignment (CBA) score, we conduct several experiments on these concept manifolds.

Figure 2: Concept Quality Across Layers: The quality of the concepts extracted from each model is shown across the network depth. For each model, concepts are extracted and evaluated at every layer. (Left) Coverage: the fraction of non-noise tokens. (Middle) Density-based clustering validation (DBCV), which assesses cluster separation. (Right) Embedding quality (NRMSE): the raw normalized root-mean-square error between pairwise distances in the activation space and the reduced embedding.

Method	NN Alignment Score $\uparrow$	Neighbor $z$ $\uparrow$
PCA (512-d)	0.983 $\pm$ 0.025	1.867 $\pm$ 0.244
CKA (linear)	0.978 $\pm$ 0.029	1.287 $\pm$ 0.486
CKA (RBF)	1.000 $\pm$ 0.000	1.714 $\pm$ 0.182
CBA (NLMCD)	0.997 $\pm$ 0.009	2.515 $\pm$ 0.256

Table 1: Nearest-neighbor alignment score and best-neighbor z-score averaged equally over 9 model checkpoints (mean

\pm

population standard deviation). All four methods recover adjacent-layer correspondence almost perfectly, so the NN alignment score alone provides limited discrimination between them. Neighbor

z

measures the strongest neighbor relative to the row mean and standard deviation. Per-model results are reported in App. Tab. 4 .

Figure 3: Qualitative comparison of alignment plots for Qwen3 8B

Panel of Figure 4 — Figure 4: CBA layer alignment. Shared color scale $[0.60,0.85]$ ; self-alignment diagonals masked. Layer indices are zero-based. All panels use 5,000 samples. Refer to the first subplot in Fig. 3 for the Qwen3 8B result and to Fig. 9 for an extended plot comparing all models.

Panel of Figure 5 — Figure 5: English concept categories and evidence orientation over residual depth. Bars show concept fractions (left axis). The dashed black line shows the total number of annotated concepts at each layer (right axis). Analogous plots for Llama 3.1 8B are shown in Fig. 10 .

Model	Paired mass	Shuffled mass	Mean excess	Peak layer	Peak excess
GPT-2	0.141	0.140	0.0009	5	0.0039
Llama-3.1-8B-Instruct	0.176	0.152	0.0242	2	0.0980
Qwen3-4B	0.245	0.196	0.0487	7	0.1187
Qwen3-8B	0.244	0.201	0.0430	8	0.1012
Qwen3-32B	0.286	0.247	0.0392	10	0.0817

Table 2: English–Mandarin shared concept mass and shuffled-pair controls over residual layers; peak layer (zero-based) indicates the layer with the largest excess with excess indicated in peak excess.

Figure 6: Semantic, syntactic, mixed, and unlabeled shared-concept mass above shuffled-pair controls for Llama-3.1-8B-Instruct, Qwen3-8B, and Qwen3-32B. Negative excess is retained; all panels share the same vertical scale.

Panel of Figure 7 — Figure 7: CBA Similarity Scores between models. Left: Qwen 4b vs 8b, Center: Qwen 8b vs. 32b, Right: Llama 8b instruct vs. Qwen 8b

Figure 8: Pairwise same-layer CBA alignment across the Llama-3.1-8B base model and the SFT, DPO, and final RLVR checkpoints of Tulu-3.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Release	HF identifier	Result section(s)
GPT-2 (124M)	02/2019	openai-community/gpt2	0, 1, 4
Llama-3.1-8B-Instruct	07/2024	meta-llama/Llama-3.1-8B-Instruct	0, 1, 2, 3, 4, 5
Qwen3-4B	05/2025	Qwen/Qwen3-4B	0, 1, 2, 4, 5
Qwen3-8B	05/2025	Qwen/Qwen3-8B	0, 1, 2, 3, 4, 5
Qwen3-32B	05/2025	Qwen/Qwen3-32B	0, 1, 2, 4
Llama-3.1-8B (base)	06/2024	meta-llama/Llama-3.1-8B	1, 6

Appendix

Table 3: Model suite used in this work.

	PCA (512-d)		CKA (linear)		CKA (RBF)		CBA (NLMCD)
Model	NN $\uparrow$	$z$ $\uparrow$	NN $\uparrow$	$z$ $\uparrow$	NN $\uparrow$	$z$ $\uparrow$	NN $\uparrow$	$z$ $\uparrow$
GPT-2	1.000	1.407	0.917	0.461	1.000	1.288	1.000	1.893
Llama3.1 8B Instruct	0.938	1.827	1.000	1.702	1.000	1.694	1.000	2.546
Llama3.1 8B Base	1.000	1.896	1.000	1.683	1.000	1.680	1.000	2.569
Qwen3 4B	0.944	1.749	0.944	0.911	1.000	1.787	0.972	2.481
Qwen3 8B	1.000	1.744	0.972	0.931	1.000	1.796	1.000	2.621

Appendix

Table 4: Per-model nearest-neighbor alignment score and mean best-neighbor z-score (

z

) for all nine checkpoints underlying Tab. 1 . NN measures whether the highest-scoring other layer is adjacent.

z

measures how strongly the best immediate neighbor stands above the row mean relative to its standard deviation.

Panel of Figure 9 — Figure 9: CBA layer alignment. Shared color scale $[0.60,0.85]$ ; self-alignment diagonals masked. Layer indices are zero-based. All panels use 5,000 samples.

Model	Concepts	Semantic	Syntactic	Mixed	Input	Both
Llama-3.1-8B-Instruct	9,398	45.7%	45.1%	9.1%	93.4%	6.6%
Qwen3-8B	11,496	39.8%	53.2%	7.0%	96.3%	3.7%

Appendix

Table 5: English concept annotations aggregated over residual layers. Percentages count concepts. Both denotes joint input/output evidence.

Panel of Figure 10 — Figure 10: English concept categories and evidence orientation over residual depth. Bars show concept fractions (left axis). The dashed black line shows the total number of annotated concepts at each layer (right axis).

Figure 11: Shared concept mass over normalized residual depth for English–Mandarin translation pairs and shuffled-pair controls.

Panel of Figure 12 — Figure 12: CBA Similarity Scores between finetune models. Left: Base vs. RLVR, Center: SFT vs. RLVR, Right: DPO vs. RLVR

Panel of Figure 13 — Figure 13: Pairwise-distance concept cost matrix and concept matches for SFT vs DPO. Left: Layer 14 Right: Layer 15

Layer	Type	SFT concept	DPO concept	Cost
14	Top	section delimiters	section delimiters	$1.0\!\times\!10^{-4}$
	Top	Prepositions and conjunctions	Prepositions and conjunctions	$1.1\!\times\!10^{-4}$
	Top	Month names	Month names	$1.3\!\times\!10^{-4}$
	Top	sentence connectors	sentence starters	$1.4\!\times\!10^{-4}$
	Top	sentence starter	demonstrative pronouns	$1.4\!\times\!10^{-4}$
	Imperfect	definite article	definite article	0.0048

Appendix

Table 6: Representative Hungarian matches between SFT and DPO at Layers 14 and 15. Lower cost indicates greater similarity. “Unmatched” concepts are assigned to private dummy nodes. For all experiments the threshold for assigning matches to dummy nodes was set to 0.0025.

Layer	Type	Llama concept	Tulu concept	Cost
0	Top	Preposition in	Preposition in	$6.0\!\times\!10^{-5}$
	Top	Equality sign	Punctuation and digits	$7.0\!\times\!10^{-5}$
	Top	Temporal conjunctions	Temporal conjunctions	$7.0\!\times\!10^{-5}$
	Imperfect	Punctuation comma	Sentence boundary	0.0044
	Imperfect	Definite article	Sentence start	0.0041
	Imperfect	Past-tense verb	Past-tense verb	0.0040

Appendix

Table 7: Representative Hungarian matches between Llama-3.1-8B-Base and Tulu-3-8B-Final at early, middle, and deep layers. Lower cost indicates greater similarity. “Unmatched” concepts are assigned to dummy nodes. For all experiments the threshold for assigning matches to dummy nodes was set to 0.0025.

Explore similar work

Jul 12, 2026cs.AI

Laguerre Geometry for Interpreting Large Language Models

Existing hypotheses represent a concept in an LLM as a single point, a linear direction, or a Gaussian cluster, yet it remains unclear how and why such structures emerge. Here, we show that concept geometry can be precisely characterized via Laguerre Geometry, in which a concept is defined as a region--a Laguerre-Voronoi cell or a union of cells--allowing us to strictly define, measure, and separate concepts. Building on this formulation, we show that finer-grained concept structures, such as inclusion and hierarchy, are naturally revealed by the Laguerre weights. We then push this geometry inside the transformer. Decomposing each layer into piecewise-linear operators, we show that a token's hidden trajectory is governed by two coupled mechanisms: a static tree of self-contained piecewise-linear flow, and a dynamic transport that hops the trajectory across trees when cross-token attention fires. This decomposition yields Geometric Lens, a training-free, hyperparameter-free method for reading out the exact concept a hidden vector encodes at any layer. We also develop Laguerre Autoencoder, a 2D visualizer that renders both the decision geometry and a model's full reasoning trajectory in one view. Finally, we move beyond explanatory geometry toward actionable interpretability, showing that Geometric Lens recovers the correct factual token when a model is prompted with in-context interference. The code is available on GitHub.

Chunwei Ma, Russell Wolfinger

JMP Statistical Discovery, Cary, NC, USA

Autoencoder Architectures Invertible Flows

Ask PDF

Aug 3, 2026cs.CL

Semantic Alignment of AI Models: Concept Collapse, Checkpoint Dynamics, and Cross-Lingual Transfer

Language model benchmarking is a difficult task. Outcome reasoning alone does not test the model's conceptualization of language and popular open-source benchmarks are quickly saturated or ingested as training data. It is important to test the model's output, but augmenting these tests by characterizing semantic structure gives more insight to how models relate abstract concepts. However, the high dimensional embedding spaces are not easy to interpret. This work demonstrates how topological methods can be used to rigorously compare these spaces to low dimensional and interpretable baselines like ontologies and curated knowledge graphs. These multi-modal alignment tests make it possible to track model adaptations and test phrase understanding across multiple languages.

Tyler Ashoff, Jordan Rodu

Department of Statistics, University of Virginia. · Department of Statistics, University of Virginia

Large Language Model Alignment Cross-Lingual Transfer Multimodal Alignment Artificial Intelligence Models Collapse Topology+1

Ask PDF

Jul 29, 2026cs.CL

From Found to Designed: Concepts as a Design Axis for Large Language Models

Large language models (LLMs) encode rich concept-like information, but represent it implicitly through distributed statistical associations rather than as explicit, structured, compositional concepts. Consequently, concept-level structure is typically \emph{found} rather than \emph{designed}: it is recovered after training through probing or dictionary learning, with no architectural guarantee of stability, compositionality, controllability, or alignment with human conceptual organization. We organize concept-aware interventions along two dimensions: whether concept structure is internally induced or externally grounded, and the stage of the pipeline where it is introduced. This taxonomy reveals three broad patterns: inference-time approaches remain comparatively underexplored, related ideas have developed largely in isolation across pipeline stages, and externally grounded methods span the entire pipeline despite often being described under different terminology. Together, these observations motivate moving beyond recovering concept-like structure from trained models toward designing LLMs with explicit conceptual representations.

Chen Shani

Tel Aviv University

Cognitive Science

Ask PDF