NinaXander: Feasibility and Limits of Composing Frozen Language Models Across Architecture Families via a Shared Latent Space

Takanori Kotama; Shun-ichiro Hayashi; Daichi Mukunoki; Tetsuya Hoshino; Takahiro Katagiri

cs.CLSep 29, 2026

NinaXander: Feasibility and Limits of Composing Frozen Language Models Across Architecture Families via a Shared Latent Space

Authors: Takanori Kotama, Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino, Takahiro Katagiri

Organizations: Graduate School of Informatics Nagoya University · Information Technology Center Nagoya University

Abstract

In this paper we propose NinaXander, a series of composed language models obtained by connecting layers of frozen language models from different architecture families with a single trained shared-latent adapter. A composed model runs the first layers of one model, converts the resulting intermediate representation once with the adapter, and then runs the remaining layers of the other model. Once the adapter is trained, several composed models that connect at different layers are obtained without retraining. Using the recurrent RWKV-4-Raven-7B and the Transformer-based Tulu-Pythia-6.9b, abbreviated as RWKV and Pythia, this study examines whether frozen models from different families can be recombined post hoc. The composed models answered multiple-choice questions, and those whose generations we examined produced syntactically well-formed text. The configuration that combines the first 5 layers of Pythia with the remaining 27 layers of RWKV reduced the Transformer key-value (KV) cache by 84.4% with accuracy not significantly different from that of RWKV alone. In multiple-choice accuracy, however, no composed model matched the parent model Pythia, and language-modeling performance decreased sharply on WikiText, a corpus of Wikipedia articles outside the training domain. The correspondence between intermediate representations was also obtained in one favorable case, with a shared tokenizer, the same depth, and the same hidden width, and does not show that the models share a general semantic space.

Figures & tables

Panel of Figure 1 — Fig. 1: Shared-latent adapter. Solid lines are the self-reconstructions and the alignment that the loss includes, and dashed lines are the cross-family read-outs that the loss does not include.

Panel of Figure 2 — Fig. 2: Configurations of the RWKV $\to$ Pythia and Pythia $\to$ RWKV chimera models.

Fig. 3: Correspondence over all $32{\times}32$ layer pairs of the unstandardized residuals $h$ taken without the adapter. Left: linear CKA. Right: fitted $R^{2}$ of per-layer linear maps. Dashed lines mark same-index layers, and the two panels use different color scales.

Panel of Figure 4 — Fig. 4: Relation between the shared latent space and the four read-outs. Read-outs of the same color share a decoder.

	ARC				SciQ
	R $\to$ P		P $\to$ R		R $\to$ P		P $\to$ R
$L$	Ad	Af	Ad	Af	Ad	Af	Ad	Af
4	59.6	41.9	62.2	60.2	70.9	47.5	69.2	69.6
8	59.0	43.1	57.3	56.0	67.7	49.1	64.6	68.0
16	48.6	34.8	57.5	57.5	54.5	35.9	68.0	69.6
24	47.0	39.4	59.6	59.2	51.1	42.7	69.9	68.2

TABLE I: Length-normalized accuracy (%) of the adapter (Ad) and of the affine control for accuracy evaluation (Af). R

\to

P: RWKV

\to

Pythia; P

\to

R: Pythia

\to

RWKV.

	ARC		SciQ
$L$	R $\to$ P	P $\to$ R	R $\to$ P	P $\to$ R
4	$59.6\!\to\!31.9$	$62.2\!\to\!31.8$	$70.9\!\to\!31.3$	$69.2\!\to\!32.6$
8	$59.0\!\to\!29.9$	$57.3\!\to\!35.5$	$67.7\!\to\!31.4$	$64.6\!\to\!37.0$
16	$48.6\!\to\!28.5$	$57.5\!\to\!43.7$	$54.5\!\to\!31.7$	$68.0\!\to\!47.3$
24	$47.0\!\to\!34.8$	$59.6\!\to\!48.4$	$51.1\!\to\!40.0$	$69.9\!\to\!49.1$

TABLE II: Change in length-normalized accuracy (%) when the nonlinear part is disabled,

\alpha:1\!\to\!0

. R

\to

P: RWKV

\to

Pythia; P

\to

R: Pythia

\to

RWKV.

	R self		R $\to$ P		P self		P $\to$ R
$L$	ARC	SciQ	ARC	SciQ	ARC	SciQ	ARC	SciQ
4	62.8	68.5	59.6	70.9	62.2	74.8	62.2	69.2
8	59.8	66.7	59.0	67.7	62.7	74.8	57.3	64.6
16	54.8	59.8	48.6	54.5	57.4	71.1	57.5	68.0
24	52.2	57.9	47.0	51.1	59.4	70.1	59.6	69.9

TABLE III: Length-normalized accuracy (%) of the four configurations. The maximum of each row and task is in bold. R self: RWKV self-reconstruction control; P self: Pythia self-reconstruction control; R

\to

P: RWKV

\to

Pythia; P

\to

R: Pythia

\to

RWKV.

	ARC			SciQ
	vs RWKV		Dir.	vs RWKV		Dir.
$L$	R $\to$ P	P $\to$ R	P $\to$ R $-$ R $\to$ P	R $\to$ P	P $\to$ R	P $\to$ R $-$ R $\to$ P
4	$-2.2$	$+0.4$	$+2.6$	$+3.9$	$+2.2$	$-1.7$
	( $0.042$ )	( $0.66$ )	( $0.015$ )	( $0.018$ )	( $0.14$ )	( $0.28$ )
8	$-2.8$	$-4.5$	$-1.7$	$+0.7$	$-2.4$	$-3.1$
	( $0.007$ )	( $<10^{-4}$ )	( $0.11$ )	( $0.69$ )	( $0.081$ )	( $0.052$ )

TABLE IV: Accuracy differences (points) between the main configurations and McNemar exact-test

p

-values in parentheses. R

\to

P: RWKV

\to

Pythia; P

\to

R: Pythia

\to

RWKV.

Config.	$L$	Trans- former layers	KV/token (KiB)	KV@4k (GiB)	Reduc. (%)	ARC/ SciQ
P alone	–	32	512	2.00	0	65.3/81.0
R $\to$ P	4	27	432	1.69	15.6	59.6/70.9
R $\to$ P	8	23	368	1.44	28.1	59.0/67.7
R $\to$ P	16	15	240	0.94	53.1	48.6/54.5
R $\to$ P	24	7	112	0.44	78.1	47.0/51.1
P $\to$ R	24	25	400	1.56	21.9	59.6/69.9

TABLE V: Accuracy and inference memory. KV@4k: KV cache size at context length

4096

. P alone: Pythia; R alone: RWKV; R

\to

P: RWKV

\to

Pythia; P

\to

R: Pythia

\to

RWKV.

Panel of Figure 10 — Fig. 5: SciQ accuracy and KV cache reduction of RWKV $\to$ Pythia and Pythia $\to$ RWKV. The green line connects the configurations in the non-dominated set.

			ARC		SciQ
Config.	$L$	Trans- former	Diff.	$p$	Diff.	$p$
P $\to$ R	4	5	$+35.4$	$<10^{-4}$	$+38.7$	$<10^{-4}$
R $\to$ P	24	7	$+18.5$	$<10^{-4}$	$+18.7$	$<10^{-4}$
P $\to$ R	8	9	$+27.6$	$<10^{-4}$	$+31.6$	$<10^{-4}$
R $\to$ P	16	15	$+11.2$	$<10^{-4}$	$+12.8$	$<10^{-4}$
P $\to$ R	16	17	$+16.9$	$<10^{-4}$	$+17.9$	$<10^{-4}$

TABLE VI: Differences in length-normalized accuracy (points, chimera model minus early exit) against a Pythia early exit that keeps the same number of Transformer layers, with McNemar exact-test

p

-values. Transformer: number of Transformer layers kept. R

\to

P: RWKV

\to

Pythia; P

\to

R: Pythia

\to

RWKV.

Panel of Figure 12 — Fig. 6: Perplexity on Alpaca and WikiText at context length $512$ . The vertical axis is logarithmic.

Prompt (verbatim)	Pythia alone	RWKV $\to$ Pythia ( $L{=}4$ )	RWKV alone	Pythia $\to$ RWKV ( $L{=}4$ )
### Instruction: What is the capital of France? ### Response:	“The capital of France is Paris.”	“The capital of France is Paris.” ✓	“Paris.”	“Paris.” ✓
The three primary colors are	“red, yellow, and blue. These colors are often used in combination to create a wide variety of colors.”	“red, yellow, and blue.” ✓	“red, blue, and yellow.” then drifts into a Q&A asking for green, the opposite color of red	“red, blue, and yellow.” then drifts into a Bob/Alice dialogue about green
Question: Water is made of hydrogen and Answer:	“Hydrogen is made of hydrogen and oxygen.”	“Hydrogen.” wrong	“hydrogen.” wrong	“hydrogen.” wrong
### Instruction: Explain what a black hole is in one sentence. ### Response:	“A black hole is a region of spacetime where gravity is so strong that nothing, not even light, can escape its pull.”	“A black hole is a region in space where gravity is so strong that nothing, not even light, can escape its pull.” ✓	“A black hole is a region of space where the gravitational pull is so strong that nothing, not even light, can escape.”	“A black hole is a type of celestial object that is formed when a star collapses and collapses into a black hole.” circular

TABLE VII: Generation examples of the two parent models and of RWKV

\to

Pythia and Pythia

\to

RWKV at

L{=}4

Explore similar work

Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models
Date pendingMarcus Armstrong, Navid Ayoobi, Arjun MukherjeeFrozen Language Model Feed-Forward
Department of Computer Science University of Houston Houston, TX 77204
One Adapter Pair per Model: A Universal Activation Interface for Language Models
Aug 10, 2026Su-Hyeon Kim, Jiwan Mun, Yo-Sub HanModel Activations
Department of Artificial Intelligence, Yonsei University · Department of Computer Science, Yonsei University
Draft-KV: Learning Useful Latent Communication Between Language Models
Sep 28, 2026Linquan Wu, Shichang Meng, Tianxiang Jiang +7Shared Control Keys And Value
City University of Hong Kong · University of Science and Technology of China · University of Electronic Science and Technology of China +3

Date pending, cs.LGJ/K move · Enter open · S save

Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models

Marcus Armstrong, Navid Ayoobi, Arjun Mukherjee

Department of Computer Science University of Houston Houston, TX 77204

Frozen Language Model Feed-Forward Cross-Attention Latent Space Private Signals

Open paper AskPDF arXiv

We present a feedforward graph architecture in which heterogeneous frozen large language models serve as computational nodes, communicating through a shared continuous latent space via learned linear projections. Building on recent work demonstrating geometric compatibility between independently trained LLM latent spaces~\cite{armstrong2026thinking}, we extend this finding from static two-model steering to end-to-end trainable multi-node graphs, where projection matrices are optimized jointly via backpropagation through residual stream injection hooks. Three small frozen models (Llama-3.2-1B, Qwen2.5-1.5B, Gemma-2-2B) encode the input into a shared latent space whose aggregate signal is injected into two larger frozen models (Phi-3-mini, Mistral-7B), whose representations feed a lightweight cross-attention output node. With only 17.6M trainable parameters against approximately 12B frozen, the architecture achieves 87.3% on ARC-Challenge, 82.8% on OpenBookQA, and 67.2% on MMLU, outperforming the best single constituent model by 11.4, 6.2, and 1.2 percentage points respectively, and outperforming parameter-matched learned classifiers on frozen single models by 9.1, 5.2, and 6.7 points. Gradient flow through multiple frozen model boundaries is empirically verified to be tractable, and the output node develops selective routing behavior across layer-2 nodes without explicit supervision.

Abstract

Explore similar work

Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models

One Adapter Pair per Model: A Universal Activation Interface for Language Models

Draft-KV: Learning Useful Latent Communication Between Language Models