Independently trained language models often learn compatible late-stage representations, despite differences in training objectives, architectures, and data modalities. We ask how far this compatibility extends: can a simple affine map let one model's hidden states be read directly by another model's output head? In this work, we learn affine transformations between the final hidden states of independent models and evaluate them across embedding classification, out-of-distribution detection, and autoregressive text generation. Across model pairs, we find that downstream performance is largely preserved under alignment, with linearly mapped source representations retaining both the decision boundaries and the confidence structure of a target model's classifier. Additionally, we show for the first time that linear alignment sometimes enables text generation across independently trained models, decoding one model's hidden states through another's frozen output head without fine-tuning. This capability is far from universal, and characterizing when it holds is a contribution of our work. We find that success is governed by two factors: tokenizer overlap, which correlates strongly with generation quality, and source-model scale, below which quality degrades sharply. Transfer is also asymmetric, with strong-to-weak mappings substantially outperforming weak-to-strong ones, indicating that source representational capacity, rather than the target head, is the limiting factor. Finally, we characterize where in the network this compatibility arises. A layer-wise analysis localizes this compatibility to the final one or two layers, indicating a late-stage, output-space phenomenon rather than evidence of shared reasoning.
Figures & tables
Figure 1: Text Generation via Cross-Model Linear Alignment: We learn an affine map from Qwen’s final hidden states into Llama’s feature space, enabling Qwen representations to be decoded by Llama’s token head. The resulting hybrid model combines Qwen’s embedding layer and transformer blocks with Llama’s output head, producing coherent responses without adopting either model’s identity.
Figure 2: Linear CKA similarity across embedding APIs. We compute linear CKA Kornblith et al. (2019) on vendor-provided embeddings over shared inputs from multiple datasets. CKA values range from 0.595 to 0.881, indicating substantial shared linear structure across independently trained models.
Figure 3: SVCCA of eight instruction tuned LLM combinations at 64 components. Mean and median bolded.
Target
Source
Classification Acc.
OOD AUROC
Baseline
Linear Map
OOD Dataset
Baseline
LinMap
SST-2 (50%)
Gemini
OpenAI
94.5
93.1
AGNews
0.826
0.801
Cohere
Gemini
94.4
91.7
0.819
0.870
OpenAI
Cohere
93.0
92.4
0.875
0.843
Mistral
Qwen
94.5
93.7
0.859
0.826
Table 1: Downstream performance is preserved under task-supervised linear alignment. We train a linear classifier fA(⋅) on target embeddings gA(⋅) ( Baseline ), then fit a linear map from source to target embedding space and apply fA to the mapped source embeddings ( Linear Map ). In-distribution classification accuracy and OOD detection AUROC are largely preserved across all model pairs.
Model 1
Model 2
Base
Mapped
M1
M2
1 → 2
2 → 1
Llama3-8B
Qwen2.5-7B
58
70
48
68
Gemma3-270M
Llama3-8B
22
58
20
49
Gemma3-270M
Qwen2.5-7B
22
70
21
68
Llama3.2-1B
Llama3-8B
42
58
36
58
Llama3.2-1B
Qwen2.5-7B
42
70
28
69
Table 2: Cross-model linear alignment on MMLU. Baseline shows native model accuracy; mapped shows accuracy after linearly transforming Model 1’s representations to Model 2’s head (M1 → M2), or vice versa. Mapping from stronger to weaker models preserves performance, while mapping from weaker to stronger degrades substantially.
Figure 4: Cross-model embedding similarity to native models. Each point represents an Alpaca test prompt. High-similarity pairs (upper right) produce coherent text, while low-similarity pairs (lower left) generate incoherent outputs.
Figure 5: Chat prompts and their corresponding completions for a given model pair using cross-model linear alignment.
Figure 6: Tokenizer compatibility predicts cross-model generation quality. Left: Exact token match rate across 23 model pairs ( r=0.898 , p<0.001 ). Right: Vocabulary overlap (Jaccard index) shows similar predictive power ( r=0.822 , p<0.001 ).
Layer
Qwen → Llama
Llama → Qwen
(lA,lB)
Acc.
(lA,lB)
Acc.
∼L/4
(7, 8)
0.00
(8, 7)
0.00
∼L/2
(14, 16)
0.00
(16, 14)
0.00
∼3L/4
(21, 24)
0.00
(24, 21)
0.00
h[−2]
(27, 31)
0.67
(31, 27)
0.00
h[−1]
(28, 32)
0.70
(32, 28)
0.58
Table 3: Layer-depth ablation on Qwen2.5-7B-Instruct ↔ Meta-Llama-3-8B-Instruct. Linear ridge mapper, 4,000 MMLU train questions ( ∼1.55\textscm token pairs), evaluated on 100 MMLU test questions via greedy mapped generation through the target LM head. OLS residual at machine epsilon in every row.
Full
Baseline
OOD
OOD + k -shot
Target
Source
Data
64
128
Only
64
128
TREC (17%)
Gemini
OpenAI
95.4
56.6
77.6
57.6
78.0
81.0
OpenAI
Qwen
96.4
59.0
82.8
58.4
77.6
83.6
Mistral
Qwen
96.6
59.4
82.2
65.6
83.8
87.4
MNLI (33%)
Table 4: Cross-model alignment classification accuracy (%) under distribution-mismatched alignment data. We fit the alignment map on a generic out-of-distribution corpus ( OOD Only ) and on that corpus plus k in-distribution examples ( OOD + k -shot ), then classify with the fixed classifier f(⋅) . Full Data is f(⋅) trained on the target’s full labeled set; Baseline 64/128 are classifiers trained on only the k examples. The OOD-only map uses no task labels yet recovers most of the full-data accuracy. Best in bold , second-best underlined ; ±1.5 pp across three seeds.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Test-set CKA between embeddings across Alpaca and Trivia QA datasets. Model embeddings were mean pooled at their final hidden state prior to calculating CKA with other models.
Figure 8: SVCCA of eight instruction tuned LLM combinations at 64 components. Mean and median are bolded.
Figure 9: SVCCA of four datasets on five embedding model combinations.
Figure 10: SVCCA of five embedding model combinations at 128 components (test set)
Party A
Party B
Classification Accuracy
OOD
OOD AUROC
(Target)
(Source)
Baseline
Linear Map
Dataset
Baseline
Linear Map
AG News (25%)
Gemini
OpenAI
92.6
91.6
MNLI
0.908
0.891
OpenAI
Qwen
92.6
91.9
MNLI
0.953
0.913
Cohere
Gemini
91.9
91.5
MNLI
0.875
0.885
OpenAI
Cohere
92.1
92.0
MNLI
0.881
0.880
Appendix
Table 15
Figure 11: Embedding similarities between greedy-mapped text generation representations and cross-model text generation representations.
Figure 12: Training and test loss for cross-model alignment as a function of training dataset size. Test loss plateaus around 4,000 samples, while training loss continues to increase, suggesting diminishing returns beyond this point.
Model A
Model B
Vocab Jaccard
Exact Match
LLM Judge Score
High-Quality Pairs (Score ≥ 3.5, both ≥ 2B)
Qwen2.5-7B-Instruct
Qwen2.5-14B-Instruct
1.000
1.000
4.68
Qwen2.5-7B-Instruct
Meta-Llama-3-8B-Instruct
0.643
0.925
4.47
Qwen2.5-14B-Instruct
Meta-Llama-3-8B-Instruct
0.643
0.925
4.43
Mistral-Nemo-Instruct-2407
Apertus-8B-Instruct-2509
0.999
1.000
4.35
Meta-Llama-3-8B-Instruct
Qwen2.5-14B-Instruct
0.643
0.925
4.31
Appendix
Table 5: Tokenizer Compatibility Metrics and Generation Quality for Model Pairs
Llama-3-8B → Qwen2.5-14B
Input a list of strings and generate all possible permutations of the strings. Input: a,b,c
The following are the possible permutations of the given list of strings: 1. "a", "b", "c"
2. "a", "c", "b"
3. "b", "a", "c"
4. "b", "c", "a"
5. "c", "a", "b"
Appendix
Table 6: Example prompts and corresponding responses generated using cross-model linear alignment for three model pairs.
Latent space alignment maps learned representations into compatible coordinates to enable comparison, transfer, and interoperability across models or data sources.
We investigate whether simple transformations can translate representations across heterogeneous text embedding models. Understanding how independently trained models organize semantic information is an enabler for AI-to-AI latent communication without decoding into human-readable text. Focusing on lightweight translators such as linear mappings, we test the literature hypothesis of latent universality in a realistic text setting beyond simplified benchmarks. Across nine embedding models differing in architecture, pooling strategy, and training objective, we evaluate compatibility using CKA, downstream transfer, fidelity, and retrieval. Simple translators recover meaningful shared structure and support transfer for some compatible pairs, but fail sharply for others. Compatibility depends jointly on architecture, training objective, pooling, and data distribution. Overall, the results show that heterogeneous embedding spaces are not universally related by simple mappings as often suggested in some literature.
Sid Ali Hamideche, Louis-Adrien Dufrène, Quentin Lampin +1
While Large Language Models (LLMs) rely on highly non-linear components, in this work we demonstrate that they exhibit fundamental linearity: when inputs from distinct text streams are linearly combined, the model outputs a superposition of the individual next-token distributions. We term this the Superposition Linearity Hypothesis. We provide evidence that superposition is an intrinsic property of the Transformer architecture rather than an emergent consequence of training; in fact, we observe that it tends to diminish as pretraining progresses. However, we demonstrate that linearity can be substantially restored through lightweight fine-tuning, significantly reducing the divergence between the predicted next-token distribution and the average of the individual next-token distributions. Finally, we introduce a guided decoding procedure that disentangles superposed outputs, enabling the simultaneous generation of two coherent continuations from a single forward pass.
Pavel Tikhonov, Anton Korznikov, Matvey Mikhalchuk +6
Post-trained language models often exhibit a recognizable AI-like writing style, yet its connection to post-training and internal representations remains poorly understood. In this work, we study whether post-training introduces or amplifies AI-like stylistic regularities and whether these regularities have a localized internal signature. To this end, we compare human text, base-model generations, and aligned-model generations under matched human-source prefixes. Aligned generations show lower human-corpus affinity and higher AI-detection rates than base generations, suggesting that post-training shifts generated text away from human-corpus style and toward detector-visible AI-like text. We then introduce PASTA (Post-training Alignment Signature Targeted Ablation), a training-free method that estimates a post-training alignment signature from aligned-base residual contrasts and ablates the corresponding direction during decoding. Across 11 aligned models and 6 AI-detectors, PASTA lowers the detection rate for most aligned models; this effect transfers well across detectors and is not reproduced by random directions. Qualitative analysis suggests that PASTA generations remain relevant and coherent while exhibiting greater stylistic variation. Together, these results show that AI-like stylistic effects of post-training can be measured, localized, and causally tested through activation ablation.
Aniket Anand, Janvijay Singh, Zhewei Sun +2
University of Chicago · University of Illinois at Urbana-Champaign · Toyota Technological Institute at Chicago