Thinking in Different Spaces: Domain-Specific Latent Geometry Survives Cross-Architecture Translation

Marcus Armstrong; Navid Ayoobi; Arjun Mukherjee

cs.LGMar 20, 2026

Thinking in Different Spaces: Domain-Specific Latent Geometry Survives Cross-Architecture Translation

Authors: Marcus Armstrong, Navid Ayoobi, Arjun Mukherjee

Organizations: Department of Computer Science, University of Houston, Houston, TX 77204

Abstract

We investigate whether independently trained language models converge to geometrically compatible latent representations, and whether this compatibility can be exploited to correct model behavior at inference time without any weight updates. We learn a linear projection matrix that maps activation vectors from a large teacher model into the coordinate system of a smaller student model, then intervene on the student's residual stream during generation by substituting its internal state with the translated teacher representation. Across a fully crossed experimental matrix of 20 heterogeneous teacher-student pairings spanning mixture-of-experts, dense, code-specialized, and synthetically trained architectures, the Ridge projection consistently achieves R^2 = 0.50 on verbal reasoning and R^2 = 0.40 on mathematical reasoning, collapsing to R^2 = -0.22 under permutation control and R^2 = 0.01 under L_1 regularization. Behavioral correction rates range from 14.0% to 50.0% on TruthfulQA (mean 25.2%) and from 8.5% to 43.3% on GSM8K arithmetic reasoning (mean 25.5%), demonstrating that the method generalizes across fundamentally different reasoning domains. We report a near-zero correlation between geometric alignment quality and behavioral correction rate (r = -0.07), revealing a dissociation between representation space fidelity and output space impact. Intervention strength is architecture-specific: student models exhibit characteristic sensitivity profiles that invert across domains, with the most steerable verbal student becoming the least steerable mathematical student. Finally, a double dissociation experiment conducted across all 20 model pairings confirms without exception that projection matrices collapse catastrophically when transferred across reasoning domains (mean R^2 = -3.83 in both transfer directions), establishing domain-specific subspace geometry as a universal property of LMs.

Figures & tables

Role	Model	Parameters	Type
Teacher	meta-llama/Meta-Llama-3-70B-Instruct	70B	Dense
Teacher	meta-llama/Meta-Llama-3-8B-Instruct	8B	Dense
Teacher	mistralai/Mistral-7B-Instruct-v0.3	7B	MoE
Teacher	google/gemma-2-9b-it	9B	Dense
Teacher	ibm-granite/granite-8b-code-instruct	8B	Code-specialized
Student	meta-llama/Llama-3.2-1B-Instruct	1B	Dense

Table 1: Teacher and student models comprising the experimental matrix. The Llama-70B teacher was loaded in 4-bit NF4 quantization due to hardware constraints.

Mapping Condition	Regularization	Mean $R^{2}$
Ridge Regression	$L_{2}$ (distributed)	$0.499$
Lasso Regression	$L_{1}$ (sparse)	$0.014$
Permutation Control	$L_{2}$ (shuffled)	$-0.220$

Table 2: Projection performance under different regularization constraints, averaged across all 20 model pairings.

Figure 1: Peak intervention correction rate ( $\Delta$ ) across the full $5\times 4$ teacher-student matrix. Each cell reports the maximum $\Delta$ over all layer combinations and $\alpha$ values.

Figure 2: Mean $\Delta$ as a function of $\alpha$ per student architecture, averaged across all teachers and layer combinations. The dashed line marks $\alpha=1$ . Phi-3-Mini exhibits a sharp peak at $\alpha=0.25$ with immediate collapse; Llama-1B maintains a flat response across the sweep.

Figure 3: Peak $\Delta$ across 16 $l_{T}\times l_{S}$ combinations for Granite-8B $\rightarrow$ Phi-3-Mini. Performance increases monotonically with teacher extraction depth; student injection depth shows a weaker, less consistent effect.

Figure 4: Peak intervention correction rate ( $\Delta$ ) on GSM8K across the full $5\times 4$ teacher-student matrix. Correction rates range from $8.5\%$ to $43.3\%$ with a mean of $25.5\%$ , virtually identical to the TruthfulQA mean of $25.2\%$ .

	TQA in-domain	TQA $\rightarrow$ GSM	GSM in-domain	GSM $\rightarrow$ TQA
Mean $R^{2}$	$0.343$	$-3.831$	$0.256$	$-1.729$
Min $R^{2}$	$0.278$	$-8.170$	$0.176$	$-3.227$
Max $R^{2}$	$0.446$	$-1.807$	$0.351$	$-1.081$

Table 3: Double dissociation results across all 20 model pairings. In-domain

R^{2}

reflects proper held-out evaluation on 100 test questions. Both transfer directions collapse catastrophically, with mean transfer

R^{2}

-3.83

and

-1.73

respectively. Dissociation confirmed in 20/20 pairs without exception.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Peak $R^{2}$ (Ridge regression) for each of the 20 teacher-student pairings, maximized over all 16 layer combinations. The Llama-8B $\rightarrow$ Llama-1B intra-family pairing achieves the highest alignment ( $R^{2}=0.684$ ). All pairings exceed $R^{2}=0.62$ , confirming broad affine compatibility across the experimental matrix.

Figure 6: Peak correction rate $\Delta$ across all 16 $l_{T}\times l_{S}$ combinations for the six highest-variance model pairings. Rows are ordered with deepest teacher extraction at the top. The monotonic improvement with teacher extraction depth observed in the main paper (Figure 3 ) generalizes across these pairings, while student injection depth consistently shows a weaker and less structured effect. The Granite-8B $\rightarrow$ Gemma-2B pairing shows the highest within-pair variance ( $\sigma=6.7\%$ ), with a pronounced performance gradient that makes layer selection particularly consequential for this pairing.

Figure 7: Mean correction rate $\Delta$ vs. $\alpha$ broken down by teacher architecture, with one line per student. The dashed vertical line marks $\alpha=1$ . Student sensitivity profiles are highly consistent across teachers: Phi-3-Mini peaks at $\alpha=0.25$ and collapses immediately regardless of teacher, while Qwen-1.5B consistently prefers $\alpha=0.5$ . The Mistral-7B panel is the notable exception, where Gemma-2B achieves its best performance at $\alpha=2.0$ rather than the sub-interpolation values preferred across other teachers, suggesting a specific compatibility between Mistral’s MoE representations and Gemma-2B’s residual stream that benefits from stronger intervention.

Teacher	Student	TQA	TQA $\rightarrow$ GSM	GSM	GSM $\rightarrow$ TQA
Llama-70B	Llama-1B	$0.354$	$-2.748$	$0.198$	$-1.108$
Llama-70B	Qwen-1.5B	$0.368$	$-8.133$	$0.222$	$-3.005$
Llama-70B	Gemma-2B	$0.290$	$-2.838$	$0.296$	$-1.261$
Llama-70B	Phi-3-Mini	$0.285$	$-1.807$	$0.300$	$-1.567$
Llama-8B	Llama-1B	$0.389$	$-2.389$	$0.230$	$-1.101$
Llama-8B	Qwen-1.5B	$0.372$	$-7.867$	$0.226$	$-3.037$

Appendix

Table 4: Full double dissociation results for all 20 pairings. TQA and GSM columns show held-out in-domain

R^{2}

. TQA

\rightarrow

GSM and GSM

\rightarrow

TQA show transfer

R^{2}

. Dissociation confirmed in all 20 pairs.

Figure 8: Mean correction rate $\Delta$ vs. $\alpha$ on GSM8K per student architecture, averaged across all teachers and layer combinations. All architectures collapse to near-zero for $\alpha\geq 0.8$ , in sharp contrast to TruthfulQA profiles where partial recovery occurs at higher $\alpha$ values. The dashed line marks $\alpha=1$ .

Figure 9: Alpha sensitivity profiles for TruthfulQA (left) and GSM8K (right), one line per student architecture averaged across all teachers and layer combinations. The dashed vertical line marks $\alpha=1$ . TruthfulQA profiles are architecturally heterogeneous with partial recovery at higher $\alpha$ ; GSM8K profiles converge uniformly on $\alpha=0.25$ with complete collapse beyond $\alpha=0.5$ across all architectures.

Figure 10: Peak correction rate $\Delta$ across all 16 $l_{T}\times l_{S}$ combinations for Gemma-9B $\rightarrow$ Qwen-1.5B on GSM8K. Rows are ordered with deepest teacher extraction at the top. The gradient with teacher depth is less monotonic than the TruthfulQA counterpart, with lower-depth teacher extraction performing more competitively.

Figure 11: Transfer $R^{2}$ across all 16 $l_{T}\times l_{S}$ combinations for Mistral-7B $\rightarrow$ Qwen-1.5B. Left panel: TruthfulQA-trained mapper evaluated on GSM8K. Right panel: GSM8K-trained mapper evaluated on TruthfulQA. Color intensity reflects absolute interference magnitude; all values are negative. Mid-network injection ( $l_{S}=0.50$ ) produces the strongest interference in both directions; late injection ( $l_{S}=0.90$ ) produces the weakest. Dissociation confirmed 16/16.

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 417 papers

May 22, 2026

Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoning

Large language models trained under diverse objectives and architectures have been shown to develop increasingly similar internal representations, an observation formalized as the Platonic Representation Hypothesis. Whether this representational convergence extends to the reasoning processes that operate over shared representations remains untested. We evaluate representational similarity across 16 language models from 8 families (1.5B to 72B parameters) on 800 reasoning problems spanning mathematics, science, commonsense, and truthfulness, stratifying by problem difficulty, computational stage, and causal relevance. Our analysis reveals three dissociations: a difficulty inversion, where models converge more on problems they collectively fail (Centered Kernel Alignment [CKA] = 0.897) than on those they solve (CKA = 0.830); a generation gap, where pre-decision representations align (CKA = 0.875) while post-decision representations diverge (CKA = 0.274); and epiphenomenal correctness, where shared information is decodable across models (66% transfer accuracy) but exerts minimal causal influence on predictions (1.5% to 5.5% flip rate across ablation protocols). These results indicate that representational convergence in language models reflects shared input processing constraints rather than shared reasoning strategies, with direct implications for ensemble design, interpretability transfer, and evaluations of model similarity. Code is available at https://github.com/Usama1002/convergence-without-understanding.

Muhammad Usama, Dong Eui Chang

Control Laboratory, School of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Daejeon 34141, Republic of Korea · Control Laboratory, School of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST)2026 Daejeon 34141, Republic of Korea

Representational Similarity Analysis LLM Interpretability Reasoning Consistency in Language Models Representation Geometry in Language Models Platonic Representation Hypothesis LLM Reasoning

Ask PDF

Apr 14, 2026

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs

Despite advances in multilingual capabilities, most large language models (LLMs) remain English-centric in their training and, crucially, in the reasoning traces they produce: even for non-English problems, they predominantly reason in English, leaving non-English users with intermediate reasoning they cannot follow. We address this disparity with two contributions. (a) We introduce

\texttt{ReasonXL}

, the first large-scale parallel corpus of cross-domain reasoning traces in five European languages (English, German, French, Italian, and Spanish): over 2.5M aligned samples (11B tokens) per language, each comprising prompt, reasoning trace, and final output, enabling direct supervision of language-specific reasoning. (b) Using

\texttt{ReasonXL}

, we demonstrate that LLMs can be adapted to reason entirely in a target language through a simple two-stage pipeline of supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR), matching or exceeding the base models' English-reasoning performance in two of four languages and recovering most of it in the remaining two.

Daniil Gurgurov, Tom Röhr, Sebastian von Rohrscheidt +3

German Research Center for Artificial Intelligence (DFKI), Saarland University · Berliner Hochschule für Technik

Cross-Lingual Reasoning in Language Models Multilingual Language Models

Ask PDF

Jun 2, 2026

Rotate2Think: Geometric Priming via Orthogonal Rotation to Improve Language Model Reasoning

Reasoning models achieve strong performance on challenging tasks by generating explicit intermediate reasoning traces before producing a final answer. Yet the internal structure of representation space when reasoning remains poorly understood: how do a model's hidden representations differ during thinking versus the embeddings of the input prompt, and can this structure be exploited to elicit stronger reasoning at inference time? We show that both input embeddings and thinking embeddings (mean-pooled last-layer hidden states over the prompt and reasoning trace, respectively) exhibit extremely high conicity, with all vectors clustering tightly around a single mean direction. Crucially, these mean input and thinking directions are non-collinear, with thinking embeddings occupying a geometrically distinct region of embedding space across many different models and benchmark tasks. This observation motivates casting the input-to-thinking transition as a rotation problem admitting a closed-form solution via orthogonal Procrustes analysis. We propose Rotate2Think, a training-free method that estimates this rotation from a small set of correctly solved examples and injects the resulting synthetic thinking vector between thinking delimiters at inference time, providing a geometric primer at the onset of the reasoning trace. Evaluated across multiple benchmarks and model families, Rotate2Think improves accuracy in 30 of 32 model-benchmark configurations across mathematics, science, and code tasks, and generalizes zero-shot to multimodal reasoning on MATH-Vision.

Aditya Sharma, Christopher J. Pal, Amal Zouaq

Polytechnique Montréal · Mila - Quebec AI Institute · LAMA-WeST Lab +1

Orthogonal Procrustes Alignment Neural Representation Geometry Representation Geometry in Language Models Activation Steering LLM Reasoning

Ask PDF

Abstract

Representation Geometry in Language Models

Explore similar work

Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoning

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs

Rotate2Think: Geometric Priming via Orthogonal Rotation to Improve Language Model Reasoning