Same Geometry, Opposite Noise: Transformer Magnitude Representations Lack Scalar Variability

Jon-Paul Cacioli

cs.CLApr 6, 2026

Same Geometry, Opposite Noise: Transformer Magnitude Representations Lack Scalar Variability

Authors: Jon-Paul Cacioli

Organizations: Independent Researcher, Melbourne, Australia

Abstract

Scalar variability -- the finding that representational noise scales proportionally with magnitude, producing a constant coefficient of variation -- is a hallmark of biological magnitude systems. We tested whether transformer language models exhibit this property by analysing the dispersion of hidden-state representations across carrier sentences for 26 numerical magnitudes in three 7-8B parameter models (Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, Llama-3-8B-Base; data from Cacioli, 2026). We found the opposite: representational variability decreased with magnitude along the magnitude axis (scaling exponent alpha approx -0.19; 0/16 primary layers with alpha > 0, all three models). The negative sign was consistent in full-dimensional space (alpha approx -0.04) and after sentence-identity correction (alpha approx -0.007). The anti-scalar pattern was 3-5x stronger along the magnitude axis than orthogonal dimensions, and corpus frequency strongly predicted per-magnitude variability (rho = .84). These results demonstrate that distributional learning alone is insufficient to produce scalar variability: transformers reproduce log-compressive magnitude geometry but not the constant-CV noise signature observed in biological systems.

Figures & tables

Figure 1: Representational variability as a function of numerical magnitude (log–log axes) at layer 16 of Llama-3-8B-Instruct. Left: raw $V_{\text{eucl}}$ . Right: sentence-corrected $V_{\text{residual}}$ . Red line: OLS fit. Grey dotted line: scalar prediction ( $\alpha=1$ ). Data cluster near constant or slightly declining variability, far below the scalar prediction.

Model	$\boldsymbol{V_{\textbf{eucl}}}$	$\boldsymbol{V_{\textbf{residual}}}$	$\boldsymbol{V_{\textbf{proj}}}$
Llama-3-8B-Instruct	$-0.046$	$-0.009$	$-0.189$
Mistral-7B-Instruct	$-0.042$	$-0.007$	$-0.210$
Llama-3-8B-Base	$-0.031$	$-0.005$	$-0.085$

Table 1: Scaling exponent

\alpha

(mean across 16 primary layers).

V_{\text{eucl}}

: mean Euclidean distance from centroid.

V_{\text{residual}}

: same, after removing sentence-level means.

V_{\text{proj}}

: SD along magnitude axis (PC1).

Figure 2: Corpus frequency predicts representational variability. Each point is one of 26 numerical magnitudes at layer 16 of Llama-3-8B-Instruct. Numbers appearing more frequently in training data show greater representational dispersion. Spearman $\rho=.882$ , $p<10^{-9}$ . Labels indicate selected magnitudes.

Figure 3: E4: On-axis (PC1, magnitude direction) vs off-axis scaling exponent across primary layers for Llama-3-8B-Base. The anti-scalar pattern is $\sim$ 3 $\times$ stronger along the magnitude axis (mean $\alpha=-0.169$ ) than orthogonal dimensions (mean $\alpha=-0.060$ ). Wilcoxon $p<.001$ . All models showed the same dissociation.

Figure 4: E6: Instruction tuning amplifies the anti-scalar pattern. Left: layerwise $\alpha$ for Llama-Instruct (mean $=-0.046$ ) vs Llama-Base (mean $=-0.031$ ). Right: per-layer difference ( $\Delta\alpha<0$ at all 16 layers, Wilcoxon $p<.001$ ).

Figure 5: Scaling exponent $\alpha$ across all layers for all three models. Left: $V_{\text{eucl}}$ (raw). Centre: $V_{\text{residual}}$ (sentence-corrected). Right: $V_{\text{proj}}$ (magnitude axis). Dashed grey line: scalar prediction ( $\alpha=1$ ). Dotted grey line: $\alpha=0$ . All models show $\alpha<0$ at all primary layers across all measures.

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 417 papers

Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models
Mar 21, 2026Jon-Paul CacioliRepresentation Geometry in Language Models Numerical Reasoning in Language Models
Independent Researcher, Melbourne Australia
Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models
Jun 1, 2026Alexander GuhaFeature Superposition Transformer Expressivity
Arizona State University
Anatomical Heterogeneity in Transformer Language Models
Mar 19, 2026Tomasz WietrzykowskiTransformer Efficient Language Model Training
Independent Researcher, Wroclaw, Poland

Mar 21, 2026J/K move · Enter open · S save

Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models

Jon-Paul Cacioli

Independent Researcher, Melbourne Australia

Representation Geometry in Language Models Numerical Reasoning in Language Models

Open paper AskPDF arXiv

How do transformer language models represent magnitude? Recent work disagrees: some find logarithmic spacing, others linear encoding, others per-digit circular representations. We apply the formal tools of psychophysics to resolve this. Using four converging paradigms (representational similarity analysis, behavioural discrimination, precision gradients, causal intervention) across three magnitude domains in three 7-9B instruction-tuned models spanning three architecture families (Llama, Mistral, Qwen), we report three findings. First, representational geometry is consistently log-compressive: RSA correlations with a Weber-law dissimilarity matrix ranged from .68 to .96 across all 96 model-domain-layer cells, with linear geometry never preferred. Second, this geometry is dissociated from behaviour: one model produces a human-range Weber fraction (WF = 0.20) while the other does not, and both models perform at chance on temporal and spatial discrimination despite possessing logarithmic geometry. Third, causal intervention reveals a layer dissociation: early layers are functionally implicated in magnitude processing (4.1x specificity) while later layers where geometry is strongest are not causally engaged (1.2x). Corpus analysis confirms the efficient coding precondition (alpha = 0.77). These results suggest that training data statistics alone are sufficient to produce log-compressive magnitude geometry, but geometry alone does not guarantee behavioural competence.

Abstract

Representation Geometry in Language Models

Explore similar work

Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models

Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models

Anatomical Heterogeneity in Transformer Language Models