Despite the continuous research and evolution of language models, they sometimes underperform previous versions. Existing approaches to overcome these challenges are resource-intensive, highlighting the need for alternatives that enable immediate action. We assume that each language model has a local module inside that is suitable for a specific function. First, this work identifies a set of modules showing consistent and local activation changes under an inference workload through activation-based analysis. Subsequently, we transplant an internal module that is properly activated for a specific task into the target model, leading to immediate and measurable functional changes without additional training or fine-tuning. To experimentally demonstrate the effectiveness of the transplant technique, we quantify the relationship between transplant strength and performance improvement under different conditions for two language models. In the cross-generation setting, we find that transplanting activation-selected modules can substantially improve the underperforming model, reaching up to twice the target baseline and achieving gap-based recovery above 100%. Moreover, in transplant experiments between a base model and its instruction-tuned counterpart, transplantation improves the underperforming model toward the stronger baseline, yielding up to about 2.33 times the target baseline with gap-based recovery reaching up to 100% in the best case. These results show that meaningful capacity transfer can be realized through the implantation of highly localized modules implied by language models. Overall, this work provides empirical evidence for task-localized modularity in language models and presents a new research area: model transplantation.
Figures & tables
Figure 1: Illustration of module-level transplantability in language models.
Component
Phi-3-mini-4k
Phi-3.5-mini
Gemma-2-2B
Gemma-2-2B-IT
Context Length
4,096 tokens
128K tokens
8K tokens
8K tokens
Vocabulary Size
32,064 tokens
32,064 tokens
256K tokens
256K tokens
Architecture
Decoder-only
Decoder-only
Decoder-only
Decoder-only
Training Corpus
Curated Phi-3 dataset
Expanded Phi-3.5 data
Gemma pretraining corpus
Gemma pretraining corpus + instruction tuning
Table 1: Comparison of key architectural and training characteristics across Phi and Gemma model variants.
Model Pair
Tokens
Dir.
Acc target (%)
Acc source (%)
Acc after (%)
Best K
TIR
Recovery (%)
Intervention Type
Phi-3 (A) ↔ Phi-3.5 (B)
32
B → A
6.0
8.0
12.0
16
2.00
300.0
FFN + Attention
64
A → B
16.0
20.0
20.0
16
1.25
100.0
FFN + Attention
128
A → B
16.0
30.0
26.0
64
1.625
71.4
FFN + Attention
256
A → B
34.0
34.0
36.0
8
1.059
–
FFN + Attention
512
B → A
36.0
40.0
40.0
8
1.111
100.0
FFN + Attention
Table 2: Best-performing transplantation settings between Phi-3 and Phi-3.5 across decoding token lengths.
K
FFN count
Attn count
Attn ratio
8
7
1
0.125
16
12
4
0.250
32
14
18
0.563
64
25
39
0.609
128
64
64
0.500
Table 3: Intervention composition as a function of transplantation scale K for Phi-3 ↔ Phi-3.5.
Figure 2: The TIR and Recovery of Phi-3 ↔ Phi-3.5 as a function of the number of transplanted modules K for different decoding lengths with max_new_tokens .
Model Pair
Tokens
Dir.
Acc B (%)
Acc A (%)
Acc after (%)
Best K
TIR
Recovery (%)
Intervention Type
Gemma-2-2B-IT (A) ↔ Gemma-2-2B (B)
32
B → A
10.0
6.0
10.0
128
1.667
100.0
FFN + Attention
64
A → B
8.0
8.0
8.0
32
1.000
–
FFN + Attention
128
A → B
6.0
22.0
10.0
64
1.667
25.0
FFN + Attention
256
A → B
6.0
36.0
12.0
64
2.000
20.0
FFN + Attention
512
A → B
6.0
30.0
14.0
64
2.333
33.3
FFN + Attention
Table 4: Best-performing transplantation settings between Gemma-2-2B and Gemma-2-2B-IT across decoding token lengths.
K
FFN count
Attn count
Attn ratio
8
0
8
1.000
16
2
14
0.875
32
2
30
0.938
64
7
57
0.891
128
39
89
0.695
Table 5: Intervention composition as a function of transplantation scale K for Gemma-2-2B-IT ↔ Gemma-2-2B.
Figure 3: The TIR and Recovery of Gemma-2-2B-IT ↔ Gemma-2-2B as a function of the number of transplanted modules K for different decoding lengths with max_new_tokens .
We find that language models can transfer capabilities through task-unrelated text. Post-training typically improves language models using task-specific data. Prior work on subliminal learning shows that information about these updates can pass through unrelated generations, but has largely focused on traits or preferences using extensive teacher outputs. We introduce Active Taskless Distillation (ATD), which achieves capability transfer using only a single word from the teacher per prompt. ATD probes the behavioral shadow of post-training by selecting prompts where the teacher and student's shared public ancestor is nearly indifferent between two ordinary words. A student initialized from this ancestor learns solely from the resulting prompt-word pairs, without target-task examples, teacher logits, or teacher parameters. In the primary coding experiment with Qwen2.5-1.5B, 5,664nses yield a 5.34 pp gain on HumanEval+ over an exact nuisance-matched control thadisrupts prompt-resperiments showtransfer in scientific knowledge, commonsense reasoning, and reading comprehensins across additional model generations, sizes, and families. Functional analyses show that the learned sid composable, andthat its strength tracks the teacher's update strength.
Ziyang Zhang, Yubin Jing, Yuanhao Zeng +3
Peking University · Georgia Institute of Technology · ShanghaiTech University +2
Adapting a language model to a specialized corpus means choosing which instruction-tuning tasks to train on under a fixed budget, and testing one choice costs a fine-tuning run. Common heuristics add more source tasks or pick sources similar to the target. The first assumes transfer is never negative; the second, that it is symmetric. We show that both assumptions fail: task A can help task B while B hurts A, so helpfulness is a signed property of ordered source--target pairs. We introduce the transfer map, a signed estimate of how much each source helps or hurts each held-out target. We fit the map in hundreds of fine-tuning runs on Qwen3 and Mistral models from 0.6B to 32B parameters, with all sources drawn from one corpus and no training examples from the target. The map predicts a held-out target's accuracy on unseen mixtures: recorded before those runs, its predictions have less than half the error of a mixture-agnostic baseline. The map is specific to its target and corpus but transfers across model scale: a mixture selected in advance at one size beats training on all source tasks at every other size we tested. Transfer is thus a property of the data. The map selects the tasks that help and drops the one that interferes: accuracy on the reasoning targets (causal explanation, multi-hop questions and methodological critique) rises by up to 14 percentage points over training on all source tasks.
Nima H. Siboni, Vahid Rostami
Juna.ai, Kastanienallee 32, 10435 Berlin, Germany · Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Germany
Specialized models encode task-oriented behavior, but transferring that behavior to a general language model usually requires training, distillation, or representation alignment. We study whether such ability can instead be transferred directly at the parameter level. We apply two existing training-free heterogeneous merging methods, previously shown to transfer knowledge between general language models, to specialist-to-general transfer, projecting a specialist donor into the recipient's shape and interpolating backbone parameters without gradient updates or semantic alignment. Intersection-Merge (IM) injects a prefix-aligned donor slice matching the recipient shape, while Activate-Prune-Merge (APM) uses forward-pass activation statistics to select which donor dimensions to retain before injection. Across embedding, reranking, reward modeling, and MoE code-specialist transfer, both methods improve the general recipient, showing that simple heterogeneous merging can move capabilities across diverse specialist roles.