Capsule Lens: Locating and Tracking Concept Geometry in Model Representations

Yiming Tang; Harshvardhan Saini; Samyak Jha; Huaming Chen; Xufeng Duan; Dianbo Liu

cs.LGSep 4, 2026

Capsule Lens: Locating and Tracking Concept Geometry in Model Representations

Authors: Yiming Tang, Harshvardhan Saini, Samyak Jha, Huaming Chen, Xufeng Duan, Dianbo Liu

Organizations: Indian Institute of Technology (ISM) Dhanbad · The Chinese University of Hong Kong · National University of Singapore

Abstract

Understanding how concepts are encoded in the internal representations of machine learning models is a central problem in mechanistic interpretability, essential both for the science of deep learning and for the trustworthy deployment of increasingly capable models. Existing approaches to interpret model representations mainly map representations onto more interpretable spaces and do not directly characterize how concepts occupy representation space; various hypotheses have been proposed, but often lack of rigorous validation and largely focus on static representations. In this work, we introduce Capsule Lens, a framework that matches the region a concept occupies with a simple, trackable geometric form, a capsule, defined by several interpretable parameters, fitted in closed form to each concept's geometry and validated on held-out samples. We apply Capsule Lens in two major settings: static and dynamic representations. On static representations, we demonstrate how to locate concept geometry across various models, and how the span and norm curves uncover important geometric characteristics. On dynamic representations, we present three case studies tracking representation drifts induced by distinct training settings, CLIP pretraining, RL post-training on visual question answering, and RL post-training on mathematical reasoning. These analyses reveal qualitatively different geometric dynamics, ranging from broad network-wide restructuring in CLIP pretraining to localized and concept-specific changes in RL post-training. Our results include findings aligned with existing literature as well as novel observations. We believe Capsule Lens stands as a promising tool for locating, analyzing, and tracking concept geometry in both static and dynamic representations.

Figures & tables

Figure 1: Overview of the Capsule Lens framework. Given a model and a dataset, the framework extracts representations for a set of predefined concepts. Capsule matching fits a capsule $\mathcal{C}(a,s_{a},n_{\mathrm{low}},n_{\mathrm{high}})$ to each concept’s representation cloud in closed form. The fitted capsule parameters and the associated span curve $F_{s}(t)$ and norm curve $F_{n}(t)$ are used to analyze the static geometry of concepts. The drift metrics—axis rotation $\Delta a$ , span change $\Delta s_{a}$ , and norm-band changes $\Delta n_{\mathrm{low}},\Delta n_{\mathrm{high}},\Delta N_{b}$ —are used to track how this geometry shifts across training checkpoints.

Metric	Definition	Interpretation
$\Delta a$	$d_{\cos}(a^{(0)},a^{(t)})$	rotation of the concept axis
$\Delta s_{a}$	$s_{a}^{(t)}-s_{a}^{(0)}$	widening ( $>0$ ) or narrowing ( $<0$ ) of angular spread
$\Delta N_{b}$	$N_{b}^{(t)}-N_{b}^{(0)}$	expansion or contraction of the norm band
$\Delta n_{\mathrm{high}}$	$n_{\mathrm{high}}^{(t)}-n_{\mathrm{high}}^{(0)}$	shift of the upper norm bound
$\Delta n_{\mathrm{low}}$	$n_{\mathrm{low}}^{(t)}-n_{\mathrm{low}}^{(0)}$	shift of the lower norm bound

Table 1: Capsule drift metrics for tracking concept geometry across checkpoints. Each metric compares the capsule fitted at checkpoint

t

against the base capsule at checkpoint

0

. The axis drift

\Delta a

measures the rotation of the concept axis; the span change

\Delta s_{a}

measures the widening (

>0

) or narrowing (

<0

) of the concept’s angular spread; and the norm-band changes

\Delta N_{b}

\Delta n_{\mathrm{high}}

\Delta n_{\mathrm{low}}

measure how the magnitude interval the concept occupies expands, contracts, or shifts.

Model	Component	Type	Layers	Capsules	Cov.	Std.	Dis.	Std.
CLIP ViT-B/32	image enc.	visual	13	13,000	0.941	0.048	0.551	0.093
CLIP ViT-L/14	image enc.	visual	25	25,000	0.941	0.048	0.533	0.073
Qwen2-VL-2B	ViT	visual	32	32,000	0.946	0.045	0.504	0.013
Qwen2-VL-2B	LLM	visual	28	28,000	0.947	0.044	0.507	0.016
LLaVA-1.5-7B	ViT (patch)	visual	24	24,000	0.934	0.051	0.506	0.015
LLaVA-1.5-7B	ViT (CLS)	visual	24	24,000	0.941	0.047	0.530	0.070

Table 2: Held-out coverage and disentanglement across models and components. We report the mean and standard deviation of

\mathrm{Cov}

and

\mathrm{Dis}

across fitted capsules.

Shape	Params	Coverage	$\mathrm{Dis}$	Angular	Norm	Separable
Capsule	$d_{\mathrm{rep}}+3$	0.943	0.583	✓	✓	✓
Cone	$d_{\mathrm{rep}}+1$	0.975	0.578	✓	$\times$	$\times$
Ball	$d_{\mathrm{rep}}+1$	0.970	0.568	✓	✓	$\times$
Band	$2$	0.962	0.514	$\times$	✓	$\times$
Box	$2d_{\mathrm{rep}}$	$0.012^{*}$	0.653	✓	✓	$\times$

Table 3: Comparison with alternative geometric approximations. Held-out coverage and disentanglement

\mathrm{Dis}

under the original min/max boundary (

q=100

), averaged across concept types, model components, and representative layers. ∗ The box exhibits extremely low held-out coverage because coordinate-wise bounds generalize poorly in high-dimensional anisotropic representation spaces, a manifestation of the curse of dimensionality ( Köppen, 2000 ) .

Figure 2: Coverage curves reveal internal concept geometry beyond capsule boundaries. For stripes at layer 12 of CLIP ViT-B/32, the span distribution and $F_{s}(t)$ reveal two angular clusters, while the norm distribution and $F_{n}(t)$ vary smoothly. The fitted span $s_{a}$ captures the outer extent of the concept region, whereas the coverage curves reveal how representations populate that region.

Figure 3: Concept disentanglement across model depth. Left: visual concepts across CLIP, Qwen2-VL, and LLaVA components, showing pronounced late-layer increases in disentanglement for several visual representations. Right: textual concepts in Qwen2.5-1.5B, where disentanglement generally decreases toward later layers, most prominently for lexical concepts.

Figure 4: CLIP pretraining restructures concept geometry throughout the network. a) Axis spans widen in blocks $2$ – $8$ but contract strongly in the final block. b) Concept axes rotate substantially at all probed depth. c–d) Norm-band midpoint and width contract steadily throughout training.

Figure 5: PPO post-training concentrates geometric change in the final layer. a) Axis spans remain nearly unchanged. b) Axis drift occurs predominantly at layer $27$ . c–d) The final-layer norm band contracts rapidly and then stabilizes while axis rotation continues.

Figure 6: RLVR induces sparse, concept-specific concentration of representation geometry. Left: at the final checkpoint, most concept–layer pairs remain close to their base geometry, while a small number exhibit pronounced span contractions or norm-bound shifts. Right: for the representative concept unit_time_minute , the final-layer axis span contracts sharply late in training while norm geometry remain comparatively stable. These results show that RLVR selectively sharpens particular concept geometries at particular layers rather than comprehensively change the model.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Effect of the number of samples

m

on fitted capsules. Results are averaged over

796

visual concepts, eight model components, three representative layers per component, and three independent draws. a) Held-out coverage increases with

m

and gradually saturates as more samples are used for capsule matching. b) Axis span

s_{a}

and norm band

N_{b}

, normalized by their values at

m=100

, increase with

m

as the fitted capsule progressively covers the concept geometry.

		Coverage			$s_{a}$		$\Delta a$
Model	Component	$m{=}25$	$m{=}50$	$m{=}100$	$m{=}25$	$m{=}100$	$m{=}25$	$m{=}100$
CLIP ViT-B/32	image enc.	0.879	0.942	0.972	0.146	0.175	0.0107	0.0027
CLIP ViT-L/14	image enc.	0.882	0.943	0.972	0.214	0.265	0.0142	0.0036
Qwen2-VL-2B	ViT	0.884	0.943	0.971	0.412	0.530	0.0277	0.0072
Qwen2-VL-2B	LLM	0.894	0.947	0.973	0.398	0.478	0.0176	0.0044
LLaVA-1.5-7B	ViT (patch)	0.852	0.932	0.968	0.646	0.745	0.0739	0.0198

Appendix

Table 4: Held-out coverage, axis span, and axis dispersion across model components. Results are computed over

796

visual concepts and three representative layers per component. Axis dispersion

\Delta a

is the cosine distance between axes independently fitted from two disjoint sample sets of the same concept.

Figure 8: Capsule parameters under percentile-based boundaries. Axis span

s_{a}

and norm band

N_{b}

are normalized by their values at

q=100

and averaged across concepts. Shaded regions indicate

\pm 1

standard deviation. Lower percentiles produce progressively tighter capsule boundaries across all four concept types.

Figure 9: Held-out coverage and disentanglement under percentile-based boundaries. Held-out coverage (left) and disentanglement

\mathrm{Dis}

(right) as the percentile

q

decreases from the original min/max boundary (

q=100

) to increasingly tighter capsules. Results are averaged across concepts, with shaded regions indicating

\pm 1

standard deviation.

	$q=100$		$q=95$		$q=90$		$q=85$		$q=80$
Shape	Cov.	$\mathrm{Dis}$	Cov.	$\mathrm{Dis}$	Cov.	$\mathrm{Dis}$	Cov.	$\mathrm{Dis}$	Cov.	$\mathrm{Dis}$
Capsule	0.943	0.583	0.855	0.620	0.769	0.641	0.686	0.658	0.609	0.673
Cone	0.975	0.578	0.918	0.614	0.862	0.635	0.805	0.651	0.750	0.665
Ball	0.977	0.568	0.922	0.601	0.865	0.620	0.811	0.635	0.757	0.649
Band	0.962	0.514	0.916	0.521	0.867	0.526	0.818	0.529	0.769	0.532
Box	0.012	0.653	0.003	0.456	0.003	0.890	0.002	1.000	0.002	1.000

Appendix

Table 5: Alternative geometric approximations under percentile-based boundaries. Held-out coverage (Cov.) and disentanglement

\mathrm{Dis}

, averaged across concept types, model components, and representative layers. Lower

q

corresponds to a tighter boundary;

q=100

recovers the original min/max construction.

Figure 10: Block-11 span contraction is visible across the concept distribution. Distribution of axis-span $s_{a}$ across concepts at block 11, by training epoch. The distribution shifts downward (mean $\approx 0.60\to 0.28$ ) and narrows over training, without a separate non-contracting subpopulation appearing at any checkpoint.

Figure 11: Top-20 most drifted concepts, block 11. Total axis drift (epoch-10 $\to$ final) summed across the five probed blocks (stacked bars, one color per block), for the 20 concepts with the largest totals. Totals span a narrow range ( $\approx 2.88$ – $2.92$ ).

Figure 12: Top-20 most stable concepts, block 11. Total axis drift (epoch-10 $\to$ final) summed across the five probed blocks, for the 20 concepts with the smallest totals. Totals ( $\approx 2.36$ – $2.46$ ) are only $\approx 20\%$ below the most-drifted group in Figure 11 , indicating that the gap between the two extremes is present but modest relative to the overall drift scale.

Figure 13: Axis drift and span change at block 11, final checkpoint. Axis drift vs. span change $\Delta s_{a}$ for individual concepts, with five example concepts labeled at each of three regions of the plot: lowest span change in magnitude (blue: square , meeting , coffee , human ), highest axis drift (red: field , grazing , zebra , zebras , enclosure ), and largest span change in magnitude (green: arms , goal , salad , batter , award ). All plotted concepts fall in a narrow drift band ( $\approx 0.65$ – $0.87$ ) and have $\Delta s_{a}<0$ ( $\Delta s_{a}\in[-0.57,-0.14]$ ).

Figure 14: Axis-span distribution at layer 27 across checkpoints. Distribution of axis-span $s_{a}$ across concepts at layer 27, by PPO training step. The distribution shape is similar across all five checkpoints; the mean varies between $\approx 0.71$ and $\approx 0.74$ .

Figure 15: Top-20 most drifted concepts, layer 27. Total axis drift at step 200, summed across the five probed layers (stacked bars, one color per layer), for the 20 concepts with the largest totals. Layer 27 (top segment) accounts for most of each total.

Figure 16: Axis drift and span change at layer 27, final checkpoint. Axis drift vs. span change $\Delta s_{a}$ (step 200 minus base) for individual concepts, with five example concepts labeled at each of three regions of the plot: largest positive span change (blue: greeting , funny , running , breakfast , coach ), lowest span change in magnitude (red: bunch , open , behind , ready , red ), and largest negative span change (green: fans , electric , west , hall , police ). Axis drift for all plotted concepts falls between $0.018$ and $0.033$ .

Figure 17: Top-20 most stable concepts, layer 27. Total axis drift at step 200, summed across the five probed layers, for the 20 concepts with the smallest totals. The y-axis maximum ( $\approx 0.027$ ) is close to Figure 15 ’s ( $\approx 0.041$ ), indicating the gap between most- and least-drifted concepts is present but modest.

Figure 18: Per-concept axis drift at layer 27 across training. Rows are the top 60 concepts by mean drift, columns are PPO training steps. Color encodes cosine distance from the base axis.

Figure 19: Population-averaged concept geometry during RLVR. (a) Mean axis span across the

18

mathematical concepts remains nearly constant at layers

3

–

20

, while layer

27

shows a small late contraction. (b) Mean axis drift from the base model remains below

7\times 10^{-6}

at every layer. (c,d) Mean norm-band midpoint and width remain largely unchanged throughout training.

Figure 20: RLVR trajectories of arithmetic-operation concepts (I). The concepts op_addition , op_subtraction , op_multiplication , and op_division remain largely stable across training, with only minor changes in axis span and norm geometry.

Figure 21: RLVR trajectories of arithmetic-operation concepts (II). The concepts op_percentage , op_fraction_ratio , and op_comparison also exhibit limited geometric change, although op_percentage shows somewhat larger variation than most other operation concepts. Overall, arithmetic-operation concepts remain geometrically stable under RLVR.

Figure 22: RLVR trajectories of entity concepts. entity_people exhibits one of the clearest geometric changes outside unit_time_minute , including a layer-

27

span contraction and substantial norm-bound changes at intermediate layers. In contrast, entity_countable_object remains comparatively stable, further illustrating the concept-specific nature of RLVR-induced geometric change.

Figure 23: RLVR trajectories of unit concepts (I). The concepts unit_dollar , unit_distance , unit_weight , and unit_time_hour remain largely stable throughout training, with only small changes in axis span and norm geometry.

Figure 24: RLVR trajectories of unit concepts (II). The concepts unit_time_day , unit_time_week , unit_time_month , and unit_time_year exhibit heterogeneous but generally limited changes. Among them, unit_time_month and unit_time_year show the largest remaining layer-

27

span contractions, but both are substantially smaller than the contraction observed for unit_time_minute in the main text. This further indicates that RLVR-induced concentration is concept-specific rather than shared uniformly across the time-unit family.

Concept	Target mode	Steered generation
church	institution	“a community of all the members of Christ… authority in the Church”
church	building	“a big, old building… red door with a clock on it”
plant	factory	“a facility for processing… waste gas emissions for steel production”
plant	organism	“long, bushy stems… grow up to 10 inches tall in their garden”
state	nation	“located in the United States… population”
state	condition	“the state of a system… an evolving and dynamic one”

Appendix

Table 6: Intra-concept steering between semantic modes. Representative generations from Qwen2.5-1.5B after steering toward different modes within the same concept geometry.

Modality	Direction	Generation
Text	church $\rightarrow$ home	Base: “a place of worship… pray, sing hymns”
		Steered: “a place where people can relax and enjoy comfort…”
Text	cell $\rightarrow$ plant	Base: “a small unit of life… DNA and proteins”
		Steered: “it has its own life… doesn’t need anything else to grow”
Vision	dog $\rightarrow$ cat	Base: “two dogs resting on a cushion in a car”
		Steered: “two cats resting on a fluffy surface…”

Appendix

Table 7: Inter-concept steering across textual and visual representations. Each example is shown with its base generation followed by the generation obtained after steering toward the target concept.

Neural Network Interpretability

Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.

See all 330 papers

Jul 12, 2026

Laguerre Geometry for Interpreting Large Language Models

Existing hypotheses represent a concept in an LLM as a single point, a linear direction, or a Gaussian cluster, yet it remains unclear how and why such structures emerge. Here, we show that concept geometry can be precisely characterized via Laguerre Geometry, in which a concept is defined as a region--a Laguerre-Voronoi cell or a union of cells--allowing us to strictly define, measure, and separate concepts. Building on this formulation, we show that finer-grained concept structures, such as inclusion and hierarchy, are naturally revealed by the Laguerre weights. We then push this geometry inside the transformer. Decomposing each layer into piecewise-linear operators, we show that a token's hidden trajectory is governed by two coupled mechanisms: a static tree of self-contained piecewise-linear flow, and a dynamic transport that hops the trajectory across trees when cross-token attention fires. This decomposition yields Geometric Lens, a training-free, hyperparameter-free method for reading out the exact concept a hidden vector encodes at any layer. We also develop Laguerre Autoencoder, a 2D visualizer that renders both the decision geometry and a model's full reasoning trajectory in one view. Finally, we move beyond explanatory geometry toward actionable interpretability, showing that Geometric Lens recovers the correct factual token when a model is prompted with in-context interference. The code is available on GitHub.

Chunwei Ma, Russell Wolfinger

JMP Statistical Discovery, Cary, NC, USA

Transformer Interpretability LLM Interpretability Representation Geometry in Language Models

Ask PDF

Jul 5, 2026

Language Models Represent and Transform Concepts with Shared Geometry

How concepts are represented in neural networks is a fundamental question in machine learning. The dominant view treats concept representations as stationary geometric objects. Yet concepts appear in context, and context transforms them. Drawing from neural population geometry, we formalize concept representations as point-cloud manifolds and contextual transformations as vector fields, and instantiate this framework in large language models. Across six model families of varying scales, we find that context moves each concept differently. The variance in these displacements is semantically organized, correlating with lexical concreteness and density. Importantly, both the concepts being transformed and this variance structure are shared across models: displacement structure transported from one model predicts held-out displacements in others significantly above chance. Together, these findings show that models share a common geometry not only in how concepts are represented, but more importantly in how context transforms them, a structure with richer organization than prior work has recognized.

Zhimin Hu, Lanhao Niu, Sashank Varma

1Georgia Institute of Technology · University of Edinburgh

Neural Representation Geometry Representation Geometry in Language Models

Ask PDF

Oct 13, 2026

What Makes a Good Visual Representation for Spatial Understanding in MLLMs?

Spatial understanding is a necessary step in moving multimodal large language models (MLLMs) toward reasoning about the 3D physical world. Their visual pretraining data, however, is almost entirely 2D, so the visual tokens processed by MLLMs often lack inherent 3D structural information. A growing number of methods inject such structure by transferring representations from external vision or geometry models, and collectively establish that such transfer helps. Behind these methods lies a largely underexplored question: what makes a visual representation good for spatial understanding? To answer it, we first separate what is transferred from how it is transferred, and then trace the resulting differences back to properties of the representations themselves. What we transfer comes from 3D geometry models, self-supervised image and video models, vision-language models, and generative models, in the form of latent features or explicit depth and point maps. We transfer it either by distillation, using it as a supervision target, or by fusion, feeding it as an additional input. We find that a representation's 3D structure correlates strongly with how much it helps, measured by cross-view consistency for latent features and by depth and reconstruction accuracy for explicit geometry. We further show that the form of a representation dictates the mechanism it needs. Latent features gain as much from fusion as from distillation; explicit geometry gains little from fusion, and most of its distillation gain depends on a geometry decoding head. Guided by these findings, we distill both latent features and explicit geometry into a single MLLM. Taking only RGB video as input, it consistently improves over its base model on both static 3D and dynamic 4D spatial reasoning benchmarks.

Xiaohu Huang, Yumin Guo, Kai Han

The University of Hong Kong

Multimodal Spatial Reasoning Multimodal Large Language Models 3D Spatial Reasoning Visual Representation Learning

Ask PDF

Abstract

Neural Network Interpretability

Explore similar work

Laguerre Geometry for Interpreting Large Language Models

Language Models Represent and Transform Concepts with Shared Geometry

What Makes a Good Visual Representation for Spatial Understanding in MLLMs?