Categorical Perception in Large Language Model Hidden States: Structural Warping at Digit-Count Boundaries

Jon-Paul Cacioli

cs.CLMar 30, 2026

Categorical Perception in Large Language Model Hidden States: Structural Warping at Digit-Count Boundaries

Authors: Jon-Paul Cacioli

Organizations: Independent Researcher, Melbourne, Australia

Abstract

Categorical perception (CP) -- enhanced discriminability at category boundaries -- is among the most studied phenomena in perceptual psychology. This paper reports that analogous geometric warping occurs in the hidden-state representations of large language models (LLMs) processing Arabic numerals. Using representational similarity analysis across six models from five architecture families, the study finds that a CP-additive model (log-distance plus a boundary boost) fits the representational geometry better than a purely continuous model at 100% of primary layers in every model tested. The effect is specific to structurally defined boundaries (digit-count transitions at 10 and 100), absent at non-boundary control positions, and absent in the temperature domain where linguistic categories (hot/cold) lack a tokenisation discontinuity. Two qualitatively distinct signatures emerge: "classic CP" (Gemma, Qwen), where models both categorise explicitly and show geometric warping, and "structural CP" (Llama, Mistral, Phi), where geometry warps at the boundary but models cannot report the category distinction. This dissociation is stable across boundaries and is a property of the architecture, not the stimulus. Structural input-format discontinuities are sufficient to produce categorical perception geometry in LLMs, independently of explicit semantic category knowledge.

Figures & tables

Model	Parameters	Family	Role
Llama-3-8B-Instruct	8.0B	Meta	Primary
Mistral-7B-Instruct-v0.3	7.2B	Mistral AI	Primary
Gemma-2-9B-IT	9.2B	Google	Primary
Qwen2.5-7B-Instruct	7.6B	Alibaba	Primary
Phi-3.5-mini-instruct	3.8B	Microsoft	Primary (scale probe)
Llama-3-8B-Base	8.0B	Meta	Exploratory (instruction-tuning control)

Table 1

Figure 1: Layerwise CP-Additive advantage ( $\Delta\rho$ ) for six models at the decade-10 boundary. Shaded region: primary layers. Llama-3-8B-Base (dashed) shows comparable geometry.

Model	Primary Layers	CP > Cont	Mean $\Delta$ $\rho$	Max $\rho$ (CP-Add)	Max $\rho$ (Cont)
Llama-3-8B-Instruct	17/17	17/17	+0.023	0.940	0.921
Mistral-7B-Instruct	17/17	17/17	+0.060	0.929	0.889
Gemma-2-9B-IT	23/23	23/23	+0.063	0.890	0.834
Qwen2.5-7B-Instruct	15/15	15/15	+0.035	0.832	0.788
Phi-3.5-mini	17/17	17/17	+0.087	0.698	0.600
Llama-3-8B-Base	17/17	17/17	+0.032	0.956	0.927

Table 3

Figure 2: Empirical RDM for Llama-3-8B-Instruct at layer 16 (decade-10, cosine distance). Block structure at the 9/10 boundary visible as increased cross-boundary distances.

Model	CP > Cont (100)	Mean $\Delta$ $\rho$ (100)	Max $\rho$ (CP-Add)	Max $\rho$ (Cont)	Ctrl-150 $\Delta$ $\rho$	E4 Ratio
Llama-3-8B-Instruct	17/17	+0.319	0.676	0.442	+0.017	12.7 $\times$
Mistral-7B-Instruct	17/17	+0.268	0.323	0.136	+0.042	3.9 $\times$
Gemma-2-9B-IT	23/23	+0.279	0.394	0.260	+0.016	4.0 $\times$
Qwen2.5-7B-Instruct	15/15	+0.162	0.404	0.241	+0.012	4.6 $\times$
Phi-3.5-mini	17/17	+0.476	0.580	0.177	+0.012	4.6 $\times$
Llama-3-8B-Base	17/17	+0.305	0.619	0.338	+0.016	8.8 $\times$

Table 5

Model	Sig layers	Mean $\Delta$ R 2	Max $\Delta$ R 2
Llama-3-8B-Instruct	17/17	0.050	0.084
Mistral-7B-Instruct	17/17	0.145	0.200
Gemma-2-9B-IT	23/23	0.174	0.231
Qwen2.5-7B-Instruct	15/15	0.079	0.115
Phi-3.5-mini	17/17	0.267	0.331
Llama-3-8B-Base	17/17	0.067	0.113

Table 6

Model	Boundary at 10?	Pattern	CP Type
Llama-3-8B-Instruct	No	Gradient, never crosses 0.50	Structural CP
Mistral-7B-Instruct	No	Extreme position bias; $\approx$ 0.50 after counterbalancing	Structural CP
Gemma-2-9B-IT	Yes	Sharp step at exactly 10 (slope = 6.3)	Classic CP
Qwen2.5-7B-Instruct	Yes	Step at 11–12 (shifted 1–2 steps from structural boundary)	Classic CP
Phi-3.5-mini	No	Flat at $\approx$ 0.50; no category signal	Structural CP
Llama-3-8B-Base	N/A	No chat template; raw-logit identification uninformative	Structural CP

Table 7

Panel of Figure 3 — Figure 3: Identification functions at decade-10 boundary. Top: Gemma (classic CP, sharp step). Bottom: Llama (structural CP, flat). Both show comparable geometry (Table 1).

Model	Conf(cross)	Conf(within)	$\Delta$ Conf	Cohen’s d	p	Sig levels
Llama-3-8B-Instruct	8.39	7.77	+0.62	0.45	<.001	4/6
Mistral-7B-Instruct	10.59	10.16	+0.43	0.29	<.001	3/6
Gemma-2-9B-IT	8.63	8.44	+0.18	0.47	<.001	4/6
Qwen2.5-7B-Instruct	20.94	20.70	+0.23	0.23	.001	2/6
Phi-3.5-mini	0.005	0.005	+0.0003	0.09	.38	1/6
Llama-3-8B-Base	0.075	0.061	+0.015	0.55	<.001	4/6

Table 9

Model	Hot/cold CP wins	Hot/cold $\Delta$ $\rho$	Temp ctrl $\Delta$ $\rho$
Llama-3-8B-Instruct	0/17	$-$ 0.062	$-$ 0.062
Mistral-7B-Instruct	1/17	$-$ 0.038	$-$ 0.061
Gemma-2-9B-IT	1/23	$-$ 0.027	$-$ 0.054
Qwen2.5-7B-Instruct	4/15	$-$ 0.009	$-$ 0.063
Phi-3.5-mini	0/17	$-$ 0.039	$-$ 0.055
Llama-3-8B-Base	0/17	$-$ 0.069	$-$ 0.065

Table 10

Model	nonce_no_order $\Delta$ $\rho$	nonce_ordered $\Delta$ $\rho$	decade_10 $\Delta$ $\rho$
Llama-3-8B-Instruct	$-$ 0.010	+0.012	+0.023
Mistral-7B-Instruct	$-$ 0.002	+0.005	+0.060
Gemma-2-9B-IT	+0.004	+0.004	+0.063
Qwen2.5-7B-Instruct	$-$ 0.010	+0.017	+0.035
Phi-3.5-mini	$-$ 0.023	+0.007	+0.087
Llama-3-8B-Base	$-$ 0.008	+0.004	+0.032

Table 11

Figure 4: E10 nonce-token control: three-level hierarchy. Mean CP advantage ( $\Delta\rho$ ) for nonce (no order), nonce (ordered), and real numbers. Real numbers produce 3–10 $\times$ larger effects.

Layer	Depth	Cat	$\Delta$ conf		Rand
5	Early	0.621	0.009	70.1 $\times$	Monotonic
8	Early-primary	0.242	0.006	43.6 $\times$	Monotonic
16	Mid-primary	0.018	0.002	10.8 $\times$	Monotonic
23	Near peak RSA	0.058	0.002	34.3 $\times$	Monotonic
27	Late	0.010	0.001	12.1 $\times$	Non-monotonic

Table 13

Figure 5: Causal intervention (E5, Llama-3-8B-Instruct). (A) Dose-response: $|\Delta\mathrm{conf}|$ vs patching dose. Early layers show strong effects; late layers flat. (B) Specificity at $\alpha=1.0$ .

Hypothesis	Outcome	Key Evidence
H0 (Falsification)	Not falsified	Control positions (15, 150) show no CP
H1 (Representational warping)	Supported	CP-Add > Cont at 100% of primary layers, all 6 models
H2 (Behavioural d ′ )	Not evaluable	Ceiling accuracy (pre-registered exclusion)
H2b (McMurray strict test)	Not evaluable	No clean identification boundary in most models
H3 (Meta-d ′ )	Not evaluable	Ceiling accuracy (pre-registered exclusion)
H4 (Boundary contribution)	Supported	$\Delta$ R 2 = 0.05–0.27 at 100% of layers, all models

Table 15

Number	Digits	Llama-3	Mistral	Gemma	Qwen	Phi
9	1	1 tok	2 tok	1 tok	1 tok	2 tok
10	2	1 tok	3 tok	2 tok	2 tok	3 tok
$\Delta$ tokens		0	+1	+1	+1	+1

Table 16

Number	Digits	Llama-3	Mistral	Gemma	Qwen	Phi
99	2	1 tok	3 tok	2 tok	2 tok	3 tok
100	3	1 tok	4 tok	3 tok	3 tok	4 tok
$\Delta$ tokens		0	+1	+1	+1	+1

Table 17

Number	Llama-3 (Instruct/Base)	Gemma-2-9B	Qwen2.5-7B	Mistral-7B	Phi-3.5-mini
4	[19] (1 tok)	[235310] (1 tok)	[19] (1 tok)	[_, 4] (2 tok)	[_, 4] (2 tok)
5	[20] (1 tok)	[235308] (1 tok)	[20] (1 tok)	[_, 5] (2 tok)	[_, 5] (2 tok)
6	[21] (1 tok)	[235318] (1 tok)	[21] (1 tok)	[_, 6] (2 tok)	[_, 6] (2 tok)
7	[22] (1 tok)	[235324] (1 tok)	[22] (1 tok)	[_, 7] (2 tok)	[_, 7] (2 tok)
8	[23] (1 tok)	[235321] (1 tok)	[23] (1 tok)	[_, 8] (2 tok)	[_, 8] (2 tok)
9	[24] (1 tok)	[235315] (1 tok)	[24] (1 tok)	[_, 9] (2 tok)	[_, 9] (2 tok)

Table 18

Tokenization in Language Models

See all 43 papers

The Geometric Price of Discrete Logic: Context-driven Manifold Dynamics of Number Representations
Mar 24, 2026Long Zhang, Dai-jun Lin, Wei-neng ChenRepresentation Geometry in Language Models LLM Interpretability
School of Computer Science and Engineering, South China University of Technology, Guangzhou City, Guangdong Province, China
Geometry of Human Perceptual Domains Emerges Transiently in LLM Representations
May 27, 2026Simardeep Singh, Paras ChopraNeural Representation Geometry LLM Interpretability
Indian Institute of Technology Roorkee, Roorkee, India · Lossfunk, India
Large language models reorganize representational geometry during in-context learning
May 16, 2026Hua-Dong Xiong, Li Ji-An, Robert C. Wilson +2Neural Representation Geometry Representation Disentanglement
School of Psychological and Brain Sciences, Georgia Tech · Department of Psychology, New York University · School of Psychological and Brain Sciences, Georgia Tech 3 Center of Excellence for Computational Cognition, Georgia Tech +2

Mar 24, 2026J/K move · Enter open · S save

The Geometric Price of Discrete Logic: Context-driven Manifold Dynamics of Number Representations

Long Zhang, Dai-jun Lin, Wei-neng Chen

School of Computer Science and Engineering, South China University of Technology, Guangzhou City, Guangdong Province, China

Representation Geometry in Language Models LLM Interpretability Numerical Reasoning in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) generalize smoothly across continuous semantic spaces, yet strict logical reasoning demands the formation of discrete decision boundaries. Prevailing theories relying on linear isometric projections fail to resolve this fundamental tension. In this work, we argue that task context operates as a non-isometric dynamical operator that enforces a necessary "topological distortion." By applying Gram-Schmidt decomposition to residual-stream activations , we reveal a dual-modulation mechanism driving this process: a class-agnostic topological preservation that anchors global structure to prevent semantic collapse, and a specific algebraic divergence that directionally tears apart cross-class concepts to forge logical boundaries. We validate this geometric evolution across a gradient of tasks, from simple mapping to complex primality testing. Crucially, targeted specific vector ablation establishes a strict causal binding between this topology and model function: algebraically erasing the divergence component collapses parity classification accuracy from 100% to chance levels (38.57%). Furthermore, we uncover a three-phase layer-wise geometric dynamic and demonstrate that under social pressure prompts, models fail to generate sufficient divergence. This results in a "manifold entanglement" that geometrically explains sycophancy and hallucination. Ultimately, our findings revise the linear-isometric presumption, demonstrating that the emergence of discrete logic in LLMs is purchased at an irreducible cost of topological deformation.

Abstract

Tokenization in Language Models

Explore similar work

The Geometric Price of Discrete Logic: Context-driven Manifold Dynamics of Number Representations

Geometry of Human Perceptual Domains Emerges Transiently in LLM Representations

Large language models reorganize representational geometry during in-context learning