Independently trained language models often learn compatible late-stage representations, despite differences in training objectives, architectures, and data modalities. We ask how far this compatibility extends: can a simple affine map let one model's hidden states be read directly by another model's output head? In this work, we learn affine transformations between the final hidden states of independent models and evaluate them across embedding classification, out-of-distribution detection, and autoregressive text generation. Across model pairs, we find that downstream performance is largely preserved under alignment, with linearly mapped source representations retaining both the decision boundaries and the confidence structure of a target model's classifier. Additionally, we show for the first time that linear alignment sometimes enables text generation across independently trained models, decoding one model's hidden states through another's frozen output head without fine-tuning. This capability is far from universal, and characterizing when it holds is a contribution of our work. We find that success is governed by two factors: tokenizer overlap, which correlates strongly with generation quality, and source-model scale, below which quality degrades sharply. Transfer is also asymmetric, with strong-to-weak mappings substantially outperforming weak-to-strong ones, indicating that source representational capacity, rather than the target head, is the limiting factor. Finally, we characterize where in the network this compatibility arises. A layer-wise analysis localizes this compatibility to the final one or two layers, indicating a late-stage, output-space phenomenon rather than evidence of shared reasoning.
Figures & tables
Figure 1: Text Generation via Cross-Model Linear Alignment: We learn an affine map from Qwen’s final hidden states into Llama’s feature space, enabling Qwen representations to be decoded by Llama’s token head. The resulting hybrid model combines Qwen’s embedding layer and transformer blocks with Llama’s output head, producing coherent responses without adopting either model’s identity.
Figure 2: Linear CKA similarity across embedding APIs. We compute linear CKA Kornblith et al. (2019) on vendor-provided embeddings over shared inputs from multiple datasets. CKA values range from 0.595 to 0.881, indicating substantial shared linear structure across independently trained models.
Figure 3: SVCCA of eight instruction tuned LLM combinations at 64 components. Mean and median bolded.
Target
Source
Classification Acc.
OOD AUROC
Baseline
Linear Map
OOD Dataset
Baseline
LinMap
SST-2 (50%)
Gemini
OpenAI
94.5
93.1
AGNews
0.826
0.801
Cohere
Gemini
94.4
91.7
0.819
0.870
OpenAI
Cohere
93.0
92.4
0.875
0.843
Mistral
Qwen
94.5
93.7
0.859
0.826
Table 1: Downstream performance is preserved under task-supervised linear alignment. We train a linear classifier fA(⋅) on target embeddings gA(⋅) ( Baseline ), then fit a linear map from source to target embedding space and apply fA to the mapped source embeddings ( Linear Map ). In-distribution classification accuracy and OOD detection AUROC are largely preserved across all model pairs.
Model 1
Model 2
Base
Mapped
M1
M2
1 → 2
2 → 1
Llama3-8B
Qwen2.5-7B
58
70
48
68
Gemma3-270M
Llama3-8B
22
58
20
49
Gemma3-270M
Qwen2.5-7B
22
70
21
68
Llama3.2-1B
Llama3-8B
42
58
36
58
Llama3.2-1B
Qwen2.5-7B
42
70
28
69
Table 2: Cross-model linear alignment on MMLU. Baseline shows native model accuracy; mapped shows accuracy after linearly transforming Model 1’s representations to Model 2’s head (M1 → M2), or vice versa. Mapping from stronger to weaker models preserves performance, while mapping from weaker to stronger degrades substantially.
Figure 4: Cross-model embedding similarity to native models. Each point represents an Alpaca test prompt. High-similarity pairs (upper right) produce coherent text, while low-similarity pairs (lower left) generate incoherent outputs.
Figure 5: Chat prompts and their corresponding completions for a given model pair using cross-model linear alignment.
Figure 6: Tokenizer compatibility predicts cross-model generation quality. Left: Exact token match rate across 23 model pairs ( r=0.898 , p<0.001 ). Right: Vocabulary overlap (Jaccard index) shows similar predictive power ( r=0.822 , p<0.001 ).
Layer
Qwen → Llama
Llama → Qwen
(lA,lB)
Acc.
(lA,lB)
Acc.
∼L/4
(7, 8)
0.00
(8, 7)
0.00
∼L/2
(14, 16)
0.00
(16, 14)
0.00
∼3L/4
(21, 24)
0.00
(24, 21)
0.00
h[−2]
(27, 31)
0.67
(31, 27)
0.00
h[−1]
(28, 32)
0.70
(32, 28)
0.58
Table 3: Layer-depth ablation on Qwen2.5-7B-Instruct ↔ Meta-Llama-3-8B-Instruct. Linear ridge mapper, 4,000 MMLU train questions ( ∼1.55\textscm token pairs), evaluated on 100 MMLU test questions via greedy mapped generation through the target LM head. OLS residual at machine epsilon in every row.
Full
Baseline
OOD
OOD + k -shot
Target
Source
Data
64
128
Only
64
128
TREC (17%)
Gemini
OpenAI
95.4
56.6
77.6
57.6
78.0
81.0
OpenAI
Qwen
96.4
59.0
82.8
58.4
77.6
83.6
Mistral
Qwen
96.6
59.4
82.2
65.6
83.8
87.4
MNLI (33%)
Table 4: Cross-model alignment classification accuracy (%) under distribution-mismatched alignment data. We fit the alignment map on a generic out-of-distribution corpus ( OOD Only ) and on that corpus plus k in-distribution examples ( OOD + k -shot ), then classify with the fixed classifier f(⋅) . Full Data is f(⋅) trained on the target’s full labeled set; Baseline 64/128 are classifiers trained on only the k examples. The OOD-only map uses no task labels yet recovers most of the full-data accuracy. Best in bold , second-best underlined ; ±1.5 pp across three seeds.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Test-set CKA between embeddings across Alpaca and Trivia QA datasets. Model embeddings were mean pooled at their final hidden state prior to calculating CKA with other models.
Figure 8: SVCCA of eight instruction tuned LLM combinations at 64 components. Mean and median are bolded.
Figure 9: SVCCA of four datasets on five embedding model combinations.
Figure 10: SVCCA of five embedding model combinations at 128 components (test set)
Party A
Party B
Classification Accuracy
OOD
OOD AUROC
(Target)
(Source)
Baseline
Linear Map
Dataset
Baseline
Linear Map
AG News (25%)
Gemini
OpenAI
92.6
91.6
MNLI
0.908
0.891
OpenAI
Qwen
92.6
91.9
MNLI
0.953
0.913
Cohere
Gemini
91.9
91.5
MNLI
0.875
0.885
OpenAI
Cohere
92.1
92.0
MNLI
0.881
0.880
Appendix
Table 15
Figure 11: Embedding similarities between greedy-mapped text generation representations and cross-model text generation representations.
Figure 12: Training and test loss for cross-model alignment as a function of training dataset size. Test loss plateaus around 4,000 samples, while training loss continues to increase, suggesting diminishing returns beyond this point.
Model A
Model B
Vocab Jaccard
Exact Match
LLM Judge Score
High-Quality Pairs (Score ≥ 3.5, both ≥ 2B)
Qwen2.5-7B-Instruct
Qwen2.5-14B-Instruct
1.000
1.000
4.68
Qwen2.5-7B-Instruct
Meta-Llama-3-8B-Instruct
0.643
0.925
4.47
Qwen2.5-14B-Instruct
Meta-Llama-3-8B-Instruct
0.643
0.925
4.43
Mistral-Nemo-Instruct-2407
Apertus-8B-Instruct-2509
0.999
1.000
4.35
Meta-Llama-3-8B-Instruct
Qwen2.5-14B-Instruct
0.643
0.925
4.31
Appendix
Table 5: Tokenizer Compatibility Metrics and Generation Quality for Model Pairs
Llama-3-8B → Qwen2.5-14B
Input a list of strings and generate all possible permutations of the strings. Input: a,b,c
The following are the possible permutations of the given list of strings: 1. "a", "b", "c"
2. "a", "c", "b"
3. "b", "a", "c"
4. "b", "c", "a"
5. "c", "a", "b"
Appendix
Table 6: Example prompts and corresponding responses generated using cross-model linear alignment for three model pairs.
Latent space alignment maps learned representations into compatible coordinates to enable comparison, transfer, and interoperability across models or data sources.