Organizations: National University of Singapore, Singapore · University of Science and Technology of China, Hefei, China · Nanyang Technological University, Singapore
Large vision-language models (LVLMs) have rapidly advanced across various domains, yet they still lag behind strong text-only large language models (LLMs) on tasks that require multi-step inference and compositional decision-making. Motivated by their shared transformer architectures, we investigate whether the two model families rely on common internal computation for such inference. At the neuron level, we uncover a surprisingly large overlap: more than half of the top-activated units during multi-step inference are shared between representative LLMs and LVLMs, revealing a modality-invariant inference subspace. Through causal probing via activation amplification, we further show that these shared neurons encode consistent and interpretable concept-level effects, demonstrating their functional contribution to inference. Building on this insight, we propose Shared Neuron Low-Rank Fusion (SNRF), a parameter-efficient framework that transfers mature inference circuitry from LLMs to LVLMs. SNRF profiles cross-model activations to identify shared neurons, computes a low-rank approximation of inter-model weight differences, and injects these updates selectively within the shared-neuron subspace. This mechanism strengthens multimodal inference performance with minimal parameter changes and requires no large-scale multimodal fine-tuning. Across diverse mathematics and perception benchmarks, SNRF consistently enhances LVLM inference performance while preserving perceptual capabilities. Our results demonstrate that shared neurons form an interpretable bridge between LLMs and LVLMs, enabling low-cost transfer of inference ability into multimodal models. Our code is available at https://github.com/chenhangcuisg-code/Do-LLMs-VLMs-Share-Neurons.
Figures & tables
Figure 1: Overview of the proposed Shared Neuron Low-Rank Fusion (SNRF) framework. SNRF first performs neuron-level activation profiling to identify neurons that are consistently co-activated during inference across an LLM and an LVLM. The intersection of these units forms a shared neuron subspace, which is reused and amplified through low-rank adapters.
Figure 2: Comparison of Base and our method across five multimodal benchmarks. Our method consistently enhances mathematical performance while maintaining comparable robustness on general metrics.
Figure 3: Overlap of activated neurons across different LVLM families.
Figure 4: Distribution of shared neurons across layers and modules in Qwen2.5-VL-7B and Qwen2.5-Math-7B. A clear concentration appears in attn.k , with secondary presence in attn.v . Clustering is most evident in early layers and later layers.
Figure 5: Effect of ablating shared neurons on MathVista . Arrows indicate higher is better.
Figure 6: Word clouds of token concepts for different neurons.
Model
CoT ↑
Format ↑
Solution ↑
MME ↑
POPE ↑
ScienceQA ↑
MMMU (val ↑ )
MMMU-Pro (V ↑ )
Qwen2.5-VL-3B
52.6
61.6
50.8
1535.0
87.2
52.5
0.461
0.018
Ours
55.2 (+2.6)
57.9 (-3.7)
55.5 (+4.7)
1559.0 (+24.0)
88.0 (+0.8)
75.1 (+22.6)
0.458 (-0.003)
0.070 (+0.052)
Qwen2.5-VL-7B
48.8
68.8
37.5
1681.0
86.2
54.5
0.503
0.116
Ours
53.3 (+4.5)
68.1 (-0.7)
42.1 (+4.6)
1713.0 (+32.0)
86.5 (+0.3)
55.4 (+0.9)
0.503 (-0.000)
0.179 (+0.063)
Intern2.5-VL-4B
60.4
65.1
61.6
1670.0
90.8
97.4
0.491
0.000
Ours
60.6 (+0.2)
65.9 (+0.8)
61.3 (-0.3)
1677.0 (+7.0)
90.7 (-0.1)
97.3 (-0.1)
0.486 (-0.005)
0.186 (+0.186)
Table 1: Main results of our Shared Neuron Low-Rank Fusion (SNRF) across backbones. Numbers in parentheses for Ours denote absolute deltas vs. the corresponding baseline. Higher is better for all metrics shown.
Figure 7: Results on MathVista-TestMini (CoT/Format/Solution), MME, POPE, ScienceQA, MMMU (val) and MMMU-Pro (V) for 3B and 7B scales. Note that Qwen2.5-VL-7B-CoT-SFT, OMlab-VL-3B-Math and Qwen2.5-VL-3B-GRIT do not have corresponding 3B and 7B versions, so their results are not reported.
Baseline
CoT
Format
Solution
MME
POPE
ScienceQA
MMMU (val)
MMMU-Pro (V)
Qwen2.5-VL-7B
48.8
68.8
37.5
1681.0
86.2
54.5
0.503
0.116
+Linear
52.4
61.3
44.1
1669.0
86.4
56.4
0.450
0.165
+DARE
49.7
65.6
43.2
1677.0
86.3
58.2
0.511
0.125
+FRANK
49.4
68.3
42.7
1667.0
86.6
57.2
0.493
0.107
Intern2.5-VL-4B
60.4
65.1
61.6
1670.0
90.8
97.4
0.491
0.000
+Linear
60.2
65.8
61.8
1670.0
90.8
97.2
0.493
0.209
Table 2: Generalizability of merge strategies. Merging shared neurons—whether by our method or alternative schemes—consistently strengthens verification-style inference ( MMMU-Pro (V) ) while preserving perception and hallucination performance. Our Ours achieves the most pronounced verification gains.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Overlap of activated neurons across different LVLM families.
Figure 9: Distribution of shared neurons across layers and modules in Intern2.5-VL-4B and Qwen2.5-GRPO-3B.
Figure 10: Distribution of shared neurons across layers and modules in Idefics3-LLaMA-8B and Deepseek-LLaMA3-8B.
Model
MathVista-Cot
Format
Solution
MME
POPE
SCIQA
MMMU
MMMU-Pro (V)
Base
QwenVL-7B
48.8
68.8
37.5
1681
86.2
54.5
0.503
0.116
QwenVL-3B
52.6
61.6
50.8
1535
87.2
52.5
0.461
0.018
Intern2.5-VL-4B
60.4
65.1
61.6
1670
90.8
97.4
0.491
0.000
Deact (shared)
QwenVL-7B
0.0
0.0
0.0
0
0.0
0.0
0.000
0.000
Appendix
Table 3: Effect of ablating shared neurons across benchmarks. Removing Nshared collapses performance to zero, while random ablations of the same size only moderately reduce scores. Higher is better for all metrics.
Figure 11: Word clouds of token concepts for different neurons.
Figure 12: Math neuron ratios across text and vision models. Vision models (red) and text models (blue) are compared by their proportion of math-related neurons, showing that a substantial fraction of shared neurons are activated by mathematical concepts.
Model Name
Params
Vision Encoder / Language Backbone
Merged / Text Backbone
Checkpoint
Idefics3‐8B‐LLaMA3
8B
SigLip vision + LLaMA3 (8B)
DeepSeek-LLaMA3
HuggingFaceM4/Idefics3-8B-LLaMA3
LLaVA-Next-8B
8B
(vision encoder as used in LLaVA-NeXT) + LLaMA3 (8B)