Capsule Lens: Locating and Tracking Concept Geometry in Model Representations

Yiming Tang; Harshvardhan Saini; Samyak Jha; Huaming Chen; Xufeng Duan; Dianbo Liu

cs.LGSep 4, 2026

Capsule Lens: Locating and Tracking Concept Geometry in Model Representations

Authors: Yiming Tang, Harshvardhan Saini, Samyak Jha, Huaming Chen, Xufeng Duan, Dianbo Liu

Organizations: Indian Institute of Technology (ISM) Dhanbad · The Chinese University of Hong Kong · National University of Singapore

Abstract

Understanding how concepts are encoded in the internal representations of machine learning models is a central problem in mechanistic interpretability, essential both for the science of deep learning and for the trustworthy deployment of increasingly capable models. Existing approaches to interpret model representations mainly map representations onto more interpretable spaces and do not directly characterize how concepts occupy representation space; various hypotheses have been proposed, but often lack of rigorous validation and largely focus on static representations. In this work, we introduce Capsule Lens, a framework that matches the region a concept occupies with a simple, trackable geometric form, a capsule, defined by several interpretable parameters, fitted in closed form to each concept's geometry and validated on held-out samples. We apply Capsule Lens in two major settings: static and dynamic representations. On static representations, we demonstrate how to locate concept geometry across various models, and how the span and norm curves uncover important geometric characteristics. On dynamic representations, we present three case studies tracking representation drifts induced by distinct training settings, CLIP pretraining, RL post-training on visual question answering, and RL post-training on mathematical reasoning. These analyses reveal qualitatively different geometric dynamics, ranging from broad network-wide restructuring in CLIP pretraining to localized and concept-specific changes in RL post-training. Our results include findings aligned with existing literature as well as novel observations. We believe Capsule Lens stands as a promising tool for locating, analyzing, and tracking concept geometry in both static and dynamic representations.

Figures & tables

Figure 1: Overview of the Capsule Lens framework. Given a model and a dataset, the framework extracts representations for a set of predefined concepts. Capsule matching fits a capsule $\mathcal{C}(a,s_{a},n_{\mathrm{low}},n_{\mathrm{high}})$ to each concept’s representation cloud in closed form. The fitted capsule parameters and the associated span curve $F_{s}(t)$ and norm curve $F_{n}(t)$ are used to analyze the static geometry of concepts. The drift metrics—axis rotation $\Delta a$ , span change $\Delta s_{a}$ , and norm-band changes $\Delta n_{\mathrm{low}},\Delta n_{\mathrm{high}},\Delta N_{b}$ —are used to track how this geometry shifts across training checkpoints.

Metric	Definition	Interpretation
$\Delta a$	$d_{\cos}(a^{(0)},a^{(t)})$	rotation of the concept axis
$\Delta s_{a}$	$s_{a}^{(t)}-s_{a}^{(0)}$	widening ( $>0$ ) or narrowing ( $<0$ ) of angular spread
$\Delta N_{b}$	$N_{b}^{(t)}-N_{b}^{(0)}$	expansion or contraction of the norm band
$\Delta n_{\mathrm{high}}$	$n_{\mathrm{high}}^{(t)}-n_{\mathrm{high}}^{(0)}$	shift of the upper norm bound
$\Delta n_{\mathrm{low}}$	$n_{\mathrm{low}}^{(t)}-n_{\mathrm{low}}^{(0)}$	shift of the lower norm bound

Table 1: Capsule drift metrics for tracking concept geometry across checkpoints. Each metric compares the capsule fitted at checkpoint

t

against the base capsule at checkpoint

0

. The axis drift

\Delta a

measures the rotation of the concept axis; the span change

\Delta s_{a}

measures the widening (

>0

) or narrowing (

<0

) of the concept’s angular spread; and the norm-band changes

\Delta N_{b}

\Delta n_{\mathrm{high}}

\Delta n_{\mathrm{low}}

measure how the magnitude interval the concept occupies expands, contracts, or shifts.

Model	Component	Type	Layers	Capsules	Cov.	Std.	Dis.	Std.
CLIP ViT-B/32	image enc.	visual	13	13,000	0.941	0.048	0.551	0.093
CLIP ViT-L/14	image enc.	visual	25	25,000	0.941	0.048	0.533	0.073
Qwen2-VL-2B	ViT	visual	32	32,000	0.946	0.045	0.504	0.013
Qwen2-VL-2B	LLM	visual	28	28,000	0.947	0.044	0.507	0.016
LLaVA-1.5-7B	ViT (patch)	visual	24	24,000	0.934	0.051	0.506	0.015
LLaVA-1.5-7B	ViT (CLS)	visual	24	24,000	0.941	0.047	0.530	0.070

Table 2: Held-out coverage and disentanglement across models and components. We report the mean and standard deviation of

\mathrm{Cov}

and

\mathrm{Dis}

across fitted capsules.

Shape	Params	Coverage	$\mathrm{Dis}$	Angular	Norm	Separable
Capsule	$d_{\mathrm{rep}}+3$	0.943	0.583	✓	✓	✓
Cone	$d_{\mathrm{rep}}+1$	0.975	0.578	✓	$\times$	$\times$
Ball	$d_{\mathrm{rep}}+1$	0.970	0.568	✓	✓	$\times$
Band	$2$	0.962	0.514	$\times$	✓	$\times$
Box	$2d_{\mathrm{rep}}$	$0.012^{*}$	0.653	✓	✓	$\times$

Table 3: Comparison with alternative geometric approximations. Held-out coverage and disentanglement

\mathrm{Dis}

under the original min/max boundary (

q=100

), averaged across concept types, model components, and representative layers. ∗ The box exhibits extremely low held-out coverage because coordinate-wise bounds generalize poorly in high-dimensional anisotropic representation spaces, a manifestation of the curse of dimensionality ( Köppen, 2000 ) .

Figure 2: Coverage curves reveal internal concept geometry beyond capsule boundaries. For stripes at layer 12 of CLIP ViT-B/32, the span distribution and $F_{s}(t)$ reveal two angular clusters, while the norm distribution and $F_{n}(t)$ vary smoothly. The fitted span $s_{a}$ captures the outer extent of the concept region, whereas the coverage curves reveal how representations populate that region.

Figure 3: Concept disentanglement across model depth. Left: visual concepts across CLIP, Qwen2-VL, and LLaVA components, showing pronounced late-layer increases in disentanglement for several visual representations. Right: textual concepts in Qwen2.5-1.5B, where disentanglement generally decreases toward later layers, most prominently for lexical concepts.

Figure 4: CLIP pretraining restructures concept geometry throughout the network. a) Axis spans widen in blocks $2$ – $8$ but contract strongly in the final block. b) Concept axes rotate substantially at all probed depth. c–d) Norm-band midpoint and width contract steadily throughout training.

Figure 5: PPO post-training concentrates geometric change in the final layer. a) Axis spans remain nearly unchanged. b) Axis drift occurs predominantly at layer $27$ . c–d) The final-layer norm band contracts rapidly and then stabilizes while axis rotation continues.

Figure 6: RLVR induces sparse, concept-specific concentration of representation geometry. Left: at the final checkpoint, most concept–layer pairs remain close to their base geometry, while a small number exhibit pronounced span contractions or norm-bound shifts. Right: for the representative concept unit_time_minute , the final-layer axis span contracts sharply late in training while norm geometry remain comparatively stable. These results show that RLVR selectively sharpens particular concept geometries at particular layers rather than comprehensively change the model.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Effect of the number of samples

m

on fitted capsules. Results are averaged over

796

visual concepts, eight model components, three representative layers per component, and three independent draws. a) Held-out coverage increases with

m

and gradually saturates as more samples are used for capsule matching. b) Axis span

s_{a}

and norm band

N_{b}

, normalized by their values at

m=100

, increase with

m

as the fitted capsule progressively covers the concept geometry.

		Coverage			$s_{a}$		$\Delta a$
Model	Component	$m{=}25$	$m{=}50$	$m{=}100$	$m{=}25$	$m{=}100$	$m{=}25$	$m{=}100$
CLIP ViT-B/32	image enc.	0.879	0.942	0.972	0.146	0.175	0.0107	0.0027
CLIP ViT-L/14	image enc.	0.882	0.943	0.972	0.214	0.265	0.0142	0.0036
Qwen2-VL-2B	ViT	0.884	0.943	0.971	0.412	0.530	0.0277	0.0072
Qwen2-VL-2B	LLM	0.894	0.947	0.973	0.398	0.478	0.0176	0.0044
LLaVA-1.5-7B	ViT (patch)	0.852	0.932	0.968	0.646	0.745	0.0739	0.0198

Appendix

Table 4: Held-out coverage, axis span, and axis dispersion across model components. Results are computed over

796

visual concepts and three representative layers per component. Axis dispersion

\Delta a

is the cosine distance between axes independently fitted from two disjoint sample sets of the same concept.

Figure 8: Capsule parameters under percentile-based boundaries. Axis span

s_{a}

and norm band

N_{b}

are normalized by their values at

q=100

and averaged across concepts. Shaded regions indicate

\pm 1

standard deviation. Lower percentiles produce progressively tighter capsule boundaries across all four concept types.

Figure 9: Held-out coverage and disentanglement under percentile-based boundaries. Held-out coverage (left) and disentanglement

\mathrm{Dis}

(right) as the percentile

q

decreases from the original min/max boundary (

q=100

) to increasingly tighter capsules. Results are averaged across concepts, with shaded regions indicating

\pm 1

standard deviation.

	$q=100$		$q=95$		$q=90$		$q=85$		$q=80$
Shape	Cov.	$\mathrm{Dis}$	Cov.	$\mathrm{Dis}$	Cov.	$\mathrm{Dis}$	Cov.	$\mathrm{Dis}$	Cov.	$\mathrm{Dis}$
Capsule	0.943	0.583	0.855	0.620	0.769	0.641	0.686	0.658	0.609	0.673
Cone	0.975	0.578	0.918	0.614	0.862	0.635	0.805	0.651	0.750	0.665
Ball	0.977	0.568	0.922	0.601	0.865	0.620	0.811	0.635	0.757	0.649
Band	0.962	0.514	0.916	0.521	0.867	0.526	0.818	0.529	0.769	0.532
Box	0.012	0.653	0.003	0.456	0.003	0.890	0.002	1.000	0.002	1.000

Appendix

Table 5: Alternative geometric approximations under percentile-based boundaries. Held-out coverage (Cov.) and disentanglement

\mathrm{Dis}

, averaged across concept types, model components, and representative layers. Lower

q

corresponds to a tighter boundary;

q=100

recovers the original min/max construction.

Figure 10: Block-11 span contraction is visible across the concept distribution. Distribution of axis-span $s_{a}$ across concepts at block 11, by training epoch. The distribution shifts downward (mean $\approx 0.60\to 0.28$ ) and narrows over training, without a separate non-contracting subpopulation appearing at any checkpoint.

Figure 11: Top-20 most drifted concepts, block 11. Total axis drift (epoch-10 $\to$ final) summed across the five probed blocks (stacked bars, one color per block), for the 20 concepts with the largest totals. Totals span a narrow range ( $\approx 2.88$ – $2.92$ ).

Figure 12: Top-20 most stable concepts, block 11. Total axis drift (epoch-10 $\to$ final) summed across the five probed blocks, for the 20 concepts with the smallest totals. Totals ( $\approx 2.36$ – $2.46$ ) are only $\approx 20\%$ below the most-drifted group in Figure 11 , indicating that the gap between the two extremes is present but modest relative to the overall drift scale.

Figure 13: Axis drift and span change at block 11, final checkpoint. Axis drift vs. span change $\Delta s_{a}$ for individual concepts, with five example concepts labeled at each of three regions of the plot: lowest span change in magnitude (blue: square , meeting , coffee , human ), highest axis drift (red: field , grazing , zebra , zebras , enclosure ), and largest span change in magnitude (green: arms , goal , salad , batter , award ). All plotted concepts fall in a narrow drift band ( $\approx 0.65$ – $0.87$ ) and have $\Delta s_{a}<0$ ( $\Delta s_{a}\in[-0.57,-0.14]$ ).

Figure 14: Axis-span distribution at layer 27 across checkpoints. Distribution of axis-span $s_{a}$ across concepts at layer 27, by PPO training step. The distribution shape is similar across all five checkpoints; the mean varies between $\approx 0.71$ and $\approx 0.74$ .

Figure 15: Top-20 most drifted concepts, layer 27. Total axis drift at step 200, summed across the five probed layers (stacked bars, one color per layer), for the 20 concepts with the largest totals. Layer 27 (top segment) accounts for most of each total.

Figure 16: Axis drift and span change at layer 27, final checkpoint. Axis drift vs. span change $\Delta s_{a}$ (step 200 minus base) for individual concepts, with five example concepts labeled at each of three regions of the plot: largest positive span change (blue: greeting , funny , running , breakfast , coach ), lowest span change in magnitude (red: bunch , open , behind , ready , red ), and largest negative span change (green: fans , electric , west , hall , police ). Axis drift for all plotted concepts falls between $0.018$ and $0.033$ .

Figure 17: Top-20 most stable concepts, layer 27. Total axis drift at step 200, summed across the five probed layers, for the 20 concepts with the smallest totals. The y-axis maximum ( $\approx 0.027$ ) is close to Figure 15 ’s ( $\approx 0.041$ ), indicating the gap between most- and least-drifted concepts is present but modest.

Figure 18: Per-concept axis drift at layer 27 across training. Rows are the top 60 concepts by mean drift, columns are PPO training steps. Color encodes cosine distance from the base axis.

Figure 19: Population-averaged concept geometry during RLVR. (a) Mean axis span across the

18

mathematical concepts remains nearly constant at layers

3

–

20

, while layer

27

shows a small late contraction. (b) Mean axis drift from the base model remains below

7\times 10^{-6}

at every layer. (c,d) Mean norm-band midpoint and width remain largely unchanged throughout training.

Figure 20: RLVR trajectories of arithmetic-operation concepts (I). The concepts op_addition , op_subtraction , op_multiplication , and op_division remain largely stable across training, with only minor changes in axis span and norm geometry.

Figure 21: RLVR trajectories of arithmetic-operation concepts (II). The concepts op_percentage , op_fraction_ratio , and op_comparison also exhibit limited geometric change, although op_percentage shows somewhat larger variation than most other operation concepts. Overall, arithmetic-operation concepts remain geometrically stable under RLVR.

Figure 22: RLVR trajectories of entity concepts. entity_people exhibits one of the clearest geometric changes outside unit_time_minute , including a layer-

27

span contraction and substantial norm-bound changes at intermediate layers. In contrast, entity_countable_object remains comparatively stable, further illustrating the concept-specific nature of RLVR-induced geometric change.

Figure 23: RLVR trajectories of unit concepts (I). The concepts unit_dollar , unit_distance , unit_weight , and unit_time_hour remain largely stable throughout training, with only small changes in axis span and norm geometry.

Figure 24: RLVR trajectories of unit concepts (II). The concepts unit_time_day , unit_time_week , unit_time_month , and unit_time_year exhibit heterogeneous but generally limited changes. Among them, unit_time_month and unit_time_year show the largest remaining layer-

27

span contractions, but both are substantially smaller than the contraction observed for unit_time_minute in the main text. This further indicates that RLVR-induced concentration is concept-specific rather than shared uniformly across the time-unit family.

Concept	Target mode	Steered generation
church	institution	“a community of all the members of Christ… authority in the Church”
church	building	“a big, old building… red door with a clock on it”
plant	factory	“a facility for processing… waste gas emissions for steel production”
plant	organism	“long, bushy stems… grow up to 10 inches tall in their garden”
state	nation	“located in the United States… population”
state	condition	“the state of a system… an evolving and dynamic one”

Appendix

Table 6: Intra-concept steering between semantic modes. Representative generations from Qwen2.5-1.5B after steering toward different modes within the same concept geometry.

Modality	Direction	Generation
Text	church $\rightarrow$ home	Base: “a place of worship… pray, sing hymns”
		Steered: “a place where people can relax and enjoy comfort…”
Text	cell $\rightarrow$ plant	Base: “a small unit of life… DNA and proteins”
		Steered: “it has its own life… doesn’t need anything else to grow”
Vision	dog $\rightarrow$ cat	Base: “two dogs resting on a cushion in a car”
		Steered: “two cats resting on a fluffy surface…”

Appendix

Table 7: Inter-concept steering across textual and visual representations. Each example is shown with its base generation followed by the generation obtained after steering toward the target concept.

Neural Network Interpretability

Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.

See all 330 papers

Laguerre Geometry for Interpreting Large Language Models
Jul 12, 2026Chunwei Ma, Russell WolfingerTransformer Interpretability LLM Interpretability
JMP Statistical Discovery, Cary, NC, USA
Language Models Represent and Transform Concepts with Shared Geometry
Jul 5, 2026Zhimin Hu, Lanhao Niu, Sashank VarmaNeural Representation Geometry Representation Geometry in Language Models
1Georgia Institute of Technology · University of Edinburgh
What Makes a Good Visual Representation for Spatial Understanding in MLLMs?
Oct 13, 2026Xiaohu Huang, Yumin Guo, Kai HanMultimodal Spatial Reasoning Multimodal Large Language Models
The University of Hong Kong

Jul 12, 2026J/K move · Enter open · S save

Laguerre Geometry for Interpreting Large Language Models

Chunwei Ma, Russell Wolfinger

JMP Statistical Discovery, Cary, NC, USA

Transformer Interpretability LLM Interpretability Representation Geometry in Language Models

Open paper AskPDF arXiv

Existing hypotheses represent a concept in an LLM as a single point, a linear direction, or a Gaussian cluster, yet it remains unclear how and why such structures emerge. Here, we show that concept geometry can be precisely characterized via Laguerre Geometry, in which a concept is defined as a region--a Laguerre-Voronoi cell or a union of cells--allowing us to strictly define, measure, and separate concepts. Building on this formulation, we show that finer-grained concept structures, such as inclusion and hierarchy, are naturally revealed by the Laguerre weights. We then push this geometry inside the transformer. Decomposing each layer into piecewise-linear operators, we show that a token's hidden trajectory is governed by two coupled mechanisms: a static tree of self-contained piecewise-linear flow, and a dynamic transport that hops the trajectory across trees when cross-token attention fires. This decomposition yields Geometric Lens, a training-free, hyperparameter-free method for reading out the exact concept a hidden vector encodes at any layer. We also develop Laguerre Autoencoder, a 2D visualizer that renders both the decision geometry and a model's full reasoning trajectory in one view. Finally, we move beyond explanatory geometry toward actionable interpretability, showing that Geometric Lens recovers the correct factual token when a model is prompted with in-context interference. The code is available on GitHub.

Abstract

Neural Network Interpretability

Explore similar work

Laguerre Geometry for Interpreting Large Language Models

Language Models Represent and Transform Concepts with Shared Geometry

What Makes a Good Visual Representation for Spatial Understanding in MLLMs?