Lexical Reference Frames: Interpreting LM Residual Streams in Native Semantics

Jian Gu; Hongyu Zhang; Chunyang Chen; Aldeida Aleti

cs.CLOct 15, 2026

Lexical Reference Frames: Interpreting LM Residual Streams in Native Semantics

Authors: Jian Gu, Hongyu Zhang, Chunyang Chen, Aldeida Aleti

Organizations: Monash University · Chongqing University · Technical University of Munich

Abstract

The vocabulary of a language model is its native semantics, and the embedding and unembedding fix the meaning of a state at the two ends of the residual stream. Between them, existing methods reuse a fixed lens or treat states as a point cloud, and neither reads a state as the model does. We argue that every hidden state has a native reference frame, induced by the layers that remain and adaptive to the context. Lexical reference frames approximate it in a model-native and context-aware way, reading a state through the remaining layers with a given number of rounds of attention between positions and a typical attention output for the rest, fitting nothing to the model's output. No lens with fixed directions per layer can reach a mean cosine with the native frame's lexical readouts above the square root of their coherence, 0.30 and 0.35 at mid-depth of SmolLM3-3B and OLMo3-7B. LexFrame[1], of order 1, reaches 0.97 and 0.97, and its readings are on average within 0.16 and 0.15 nats of the model's output. Its coefficient differences attribute predictions more faithfully than attribution patching, and its lexical basis steers predictions while sparing other tokens. On average, its readings settle on the model's top token over the final 0.82 and 0.81 of the layers, those of lenses over at most 0.11. These frames thus make every hidden state readable, attributable and steerable.

Figures & tables

Figure 1: Angle between the lexical readout of LexFrame [

\infty

] for a label (grey) and the direction each method offers, after every fourth layer of OLMo3-7B, as means over states and labels, zero at the output by definition.

Figure 2: Lexical bases of five tokens (colours) and one state (black) across the layers, fixed at the two interfaces (filled) and shifting from layer to layer in between (hollow).

Figure 3: What a frame of each order computes to read the state at position

t

of layer

\ell

, with the layer’s own attention (solid black) and typical attention outputs (dashed blue). The order in brackets counts the rounds of attention between positions above layer

\ell

. The context reaches the state only through the state itself (0), directly from the prefix (1), through one other prefix position (2) or without limit (

\infty

Figure 4: Divergence in nats from the model’s output to the reading of each frame and each lens, per layer. Each further order lowers it at nearly every layer, while both lenses stay several nats above all frames until near the output.

SmolLM3-3B
Method	KL $\downarrow$	Top-1 $\uparrow$	ECE $\downarrow$
Logit lens	29.06	0.04	0.632
Jacobian lens	7.51	0.14	0.376
LexFrame [0]	0.87	0.64	0.018
LexFrame [1]	0.16	0.86	0.027
LexFrame [2]	0.06	0.92	0.020

Table 1: Layer-averaged readings: divergence from the model, top-token agreement and calibration error.

Figure 5: Share of a lexical readout’s squared norm in 64 local or global principal directions of the cloud.

Distance	SmolLM3-3B	OLMo3-7B
Logit lens	0.11	0.16
Jacobian lens	0.28	0.24
Cosine	0.27	0.28
Euclidean	0.08	0.12
Principal subspace	0.09	0.13
Mahalanobis	0.08	0.03

Table 2: Layer-averaged rank correlation between a distance of two states and the effect of their substitution.

Direction	SmolLM3-3B	OLMo3-7B
Input frame	0.42	0.00
Output frame	0.45	0.80
LexFrame [1] , readout	0.54	0.73
LexFrame [1] , basis	0.84	0.97
LexFrame [ $\infty$ ] , readout	0.56	0.79
LexFrame [ $\infty$ ] , basis	0.98	1.00

Table 3: Share of moves along the lexical readout or basis of a label that raise its logit most, averaged over layers, where fixed frames use their basis unchanged.

	SmolLM3-3B		OLMo3-7B
Paired state	KL $\downarrow$	Top-1 $\uparrow$	KL $\downarrow$	Top-1 $\uparrow$
Layer state	8.36	0.04	5.64	0.15
Fixed map	5.19	0.13	5.46	0.13
LexFrame [1]	0.16	0.87	0.15	0.84
LexFrame [ $\infty$ ]	0.00	1.00	0.00	1.00

Table 4: Coefficients on the output lexical basis against model logits on 512 positions, averaged over layers: divergence (nats) at the best scale, top-token agreement.

SmolLM3-3B
Direction	Reach $\uparrow$	Side $\downarrow$	Clean $\uparrow$
Logit lens	0.69	0.20	0.49
Jacobian lens	0.78	0.37	0.49
LexFrame [1] , readout	0.92	0.23	0.61
LexFrame [1] , basis	0.89	0.12	0.71
LexFrame [ $\infty$ ] , readout	0.95	0.22	0.63

Table 5: Steering along the lexical readout or basis, averaged over layers: share of predictions flipped, mean change in the other tokens at the first flip (nats) and share flipped with a change below 0.1 nats.

SmolLM3-3B
Attribution	Layer	Attention	MLP
Logit lens	0.08	0.13	0.04
Jacobian lens	0.12	0.18	0.10
LexFrame [1] , diff	0.88	0.79	0.88
LexFrame [1] , linear	0.57	0.67	0.58
LexFrame [ $\infty$ ] , linear	0.62	0.75	0.63

Table 6: Layer-averaged correlation of attributed and measured effects on the predicted centred logit.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Cosine between the lexical readouts of LexFrame [

\infty

] at a layer and at the last layer, from the first checkpoint (lightest) to the base model (darkest).

Figure 7: Cosine of each method’s direction for a label with the lexical readout of LexFrame [

\infty

] , per layer, and the coherence bound

\sqrt{\kappa_{\ell}}

Figure 8: Mean angle of each method’s direction to the lexical readout of LexFrame [

\infty

] per layer, with the exponential fit for LexFrame [1] and the plateau in the coherence of SmolLM3 (shaded).

Figure 9: Divergence from the model’s output to each reading by position, averaged over layers, for sequences of 1024 tokens (thick) and 128 tokens (thin).

SmolLM3-3B
Method	Settled layers $\uparrow$	Skipped layers $\uparrow$
Logit lens	0.03	0.00
Jacobian lens	0.09	0.00
LexFrame [0]	0.58	0.23
LexFrame [1]	0.82	0.45
LexFrame [2]	0.89	0.53

Appendix

Table 7: Mean share of layers, counted back from the last, over which each reading settles on the model’s top token, and mean share that a threshold-based exit skips.

SmolLM3-3B
Fitted terms	KL $\downarrow$	Cosine $\uparrow$	Attribution $\uparrow$
Regression	0.16	0.89	0.88
Mean	1.26	0.73	0.67
Zero	0.76	0.71	0.68
OLMo3-7B
Regression	0.15	0.87	0.90

Appendix

Table 8: LexFrame [1] with fitted terms (regression), mean or zero output, averaged over layers: divergence (nats), cosine with LexFrame [

\infty

] and attribution.

Layer	Logit lens	Jacobian lens	LexFrame [1]
0	“ oretical ”	–	“ New ”
4	“ oretical ”	“ flavours ”	“ Bill ”
8	“ nto ”	“ urga ”	“ Bill ”
12	“ comparison ”	“ ...\n\n ”	“ Times ”
16	“ orem ”	“ Illustrated ”	“ Guardian ”
20	“ odor ”	“ Guardian ”	“ Guardian ”

Appendix

Table 9: Top token of each reading after every fourth layer of OLMo3-7B at one position, where layer 32 is the output of the whole model.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 776 papers

SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models
Jun 30, 2026Jian Gu, Aldeida Aleti, Chunyang Chen +1Mechanistic Interpretability Representation Geometry in Language Models
Monash University · Technical University of Munich · Chongqing University
On the Persistent Effects of Lexicality in Large Language Models
Jun 1, 2026Hammad Rizwan, Muhammad Umair Haider, Nishant Subramani +3LLM Interpretability Language Modeling
Dalhousie University Halifax, NS · University of Kentucky Lexington, KY · Carnegie Mellon University Pittsburgh, PA
Everyone Was Right: Lexical Representations in Language Models Are Both Abstract and Holistic
Oct 13, 2026Zachary Nicholas Houghton, Yu Zhou, Dan Pluth +2Language Modeling Representation Geometry in Language Models
University of Oregon · Vail Systems, Inc.

Jun 30, 2026J/K move · Enter open · S save

SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models

Jian Gu, Aldeida Aleti, Chunyang Chen, Hongyu Zhang

Monash University · Technical University of Munich · Chongqing University

Mechanistic Interpretability Representation Geometry in Language Models

Open paper AskPDF arXiv

Residual-stream analysis asks how language-model computation evolves across depth, but intermediate decoding requires comparable readout coordinates across layers. If embedding anchors and unembedding readout disagree on the chosen span, apparent motion may reflect measurement drift rather than computation. We introduce Semantic Reference Frames (SemRF), an anchor-based formalism separating semantic measurement from residual dynamics. A SemRF fixes anchors and measures states against them. Pseudo-inverse tying gives exact synchronization; under restricted bi-invertibility, SemRF yields stable semantic-basis coordinates, distortion bounds, and near-identity changes. With the frame fixed, residual computation becomes a depthwise semantic trajectory. The anchors induce a semantic Voronoi diagram: distance, or evidence such as logits, assigns each layer to a coarse cell, while coordinates retain within-cell motion and margins. We define layerwise steps, contribution profiles, and imbalance diagnostics, then use the Voronoi trace to define a margin-relaxed tube. The canonical trace is the minimum-action path inside this tube; when nonempty with positive quadratic weight, it is unique and obeys a discrete spline equation away from active constraints. Excess action controls step, curvature, and profile mismatch. Low curvature implies piecewise-linear compressibility and local knowledge density: lower trace complexity means fewer semantic knots. Through the parameter-to-trajectory map, this gives a conditional link to parameter efficiency: among admissible settings fitting data, lower-action and lower-complexity traces use fewer semantic degrees of freedom. The guarantees require controlled interface error and small projection residual under explicit tube constraints.

Abstract

LLM Interpretability

Explore similar work

SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models

On the Persistent Effects of Lexicality in Large Language Models

Everyone Was Right: Lexical Representations in Language Models Are Both Abstract and Holistic