Organizations: National University of Singapore, Singapore · University of Science and Technology of China, Hefei, China · Nanyang Technological University, Singapore
Large vision-language models (LVLMs) have rapidly advanced across various domains, yet they still lag behind strong text-only large language models (LLMs) on tasks that require multi-step inference and compositional decision-making. Motivated by their shared transformer architectures, we investigate whether the two model families rely on common internal computation for such inference. At the neuron level, we uncover a surprisingly large overlap: more than half of the top-activated units during multi-step inference are shared between representative LLMs and LVLMs, revealing a modality-invariant inference subspace. Through causal probing via activation amplification, we further show that these shared neurons encode consistent and interpretable concept-level effects, demonstrating their functional contribution to inference. Building on this insight, we propose Shared Neuron Low-Rank Fusion (SNRF), a parameter-efficient framework that transfers mature inference circuitry from LLMs to LVLMs. SNRF profiles cross-model activations to identify shared neurons, computes a low-rank approximation of inter-model weight differences, and injects these updates selectively within the shared-neuron subspace. This mechanism strengthens multimodal inference performance with minimal parameter changes and requires no large-scale multimodal fine-tuning. Across diverse mathematics and perception benchmarks, SNRF consistently enhances LVLM inference performance while preserving perceptual capabilities. Our results demonstrate that shared neurons form an interpretable bridge between LLMs and LVLMs, enabling low-cost transfer of inference ability into multimodal models. Our code is available at https://github.com/chenhangcuisg-code/Do-LLMs-VLMs-Share-Neurons.
Figures & tables
Figure 1: Overview of the proposed Shared Neuron Low-Rank Fusion (SNRF) framework. SNRF first performs neuron-level activation profiling to identify neurons that are consistently co-activated during inference across an LLM and an LVLM. The intersection of these units forms a shared neuron subspace, which is reused and amplified through low-rank adapters.
Figure 2: Comparison of Base and our method across five multimodal benchmarks. Our method consistently enhances mathematical performance while maintaining comparable robustness on general metrics.
Figure 3: Overlap of activated neurons across different LVLM families.
Figure 4: Distribution of shared neurons across layers and modules in Qwen2.5-VL-7B and Qwen2.5-Math-7B. A clear concentration appears in attn.k , with secondary presence in attn.v . Clustering is most evident in early layers and later layers.
Figure 5: Effect of ablating shared neurons on MathVista . Arrows indicate higher is better.
Figure 6: Word clouds of token concepts for different neurons.
Model
CoT ↑
Format ↑
Solution ↑
MME ↑
POPE ↑
ScienceQA ↑
MMMU (val ↑ )
MMMU-Pro (V ↑ )
Qwen2.5-VL-3B
52.6
61.6
50.8
1535.0
87.2
52.5
0.461
0.018
Ours
55.2 (+2.6)
57.9 (-3.7)
55.5 (+4.7)
1559.0 (+24.0)
88.0 (+0.8)
75.1 (+22.6)
0.458 (-0.003)
0.070 (+0.052)
Qwen2.5-VL-7B
48.8
68.8
37.5
1681.0
86.2
54.5
0.503
0.116
Ours
53.3 (+4.5)
68.1 (-0.7)
42.1 (+4.6)
1713.0 (+32.0)
86.5 (+0.3)
55.4 (+0.9)
0.503 (-0.000)
0.179 (+0.063)
Intern2.5-VL-4B
60.4
65.1
61.6
1670.0
90.8
97.4
0.491
0.000
Ours
60.6 (+0.2)
65.9 (+0.8)
61.3 (-0.3)
1677.0 (+7.0)
90.7 (-0.1)
97.3 (-0.1)
0.486 (-0.005)
0.186 (+0.186)
Table 1: Main results of our Shared Neuron Low-Rank Fusion (SNRF) across backbones. Numbers in parentheses for Ours denote absolute deltas vs. the corresponding baseline. Higher is better for all metrics shown.
Figure 7: Results on MathVista-TestMini (CoT/Format/Solution), MME, POPE, ScienceQA, MMMU (val) and MMMU-Pro (V) for 3B and 7B scales. Note that Qwen2.5-VL-7B-CoT-SFT, OMlab-VL-3B-Math and Qwen2.5-VL-3B-GRIT do not have corresponding 3B and 7B versions, so their results are not reported.
Baseline
CoT
Format
Solution
MME
POPE
ScienceQA
MMMU (val)
MMMU-Pro (V)
Qwen2.5-VL-7B
48.8
68.8
37.5
1681.0
86.2
54.5
0.503
0.116
+Linear
52.4
61.3
44.1
1669.0
86.4
56.4
0.450
0.165
+DARE
49.7
65.6
43.2
1677.0
86.3
58.2
0.511
0.125
+FRANK
49.4
68.3
42.7
1667.0
86.6
57.2
0.493
0.107
Intern2.5-VL-4B
60.4
65.1
61.6
1670.0
90.8
97.4
0.491
0.000
+Linear
60.2
65.8
61.8
1670.0
90.8
97.2
0.493
0.209
Table 2: Generalizability of merge strategies. Merging shared neurons—whether by our method or alternative schemes—consistently strengthens verification-style inference ( MMMU-Pro (V) ) while preserving perception and hallucination performance. Our Ours achieves the most pronounced verification gains.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Overlap of activated neurons across different LVLM families.
Figure 9: Distribution of shared neurons across layers and modules in Intern2.5-VL-4B and Qwen2.5-GRPO-3B.
Figure 10: Distribution of shared neurons across layers and modules in Idefics3-LLaMA-8B and Deepseek-LLaMA3-8B.
Model
MathVista-Cot
Format
Solution
MME
POPE
SCIQA
MMMU
MMMU-Pro (V)
Base
QwenVL-7B
48.8
68.8
37.5
1681
86.2
54.5
0.503
0.116
QwenVL-3B
52.6
61.6
50.8
1535
87.2
52.5
0.461
0.018
Intern2.5-VL-4B
60.4
65.1
61.6
1670
90.8
97.4
0.491
0.000
Deact (shared)
QwenVL-7B
0.0
0.0
0.0
0
0.0
0.0
0.000
0.000
Appendix
Table 3: Effect of ablating shared neurons across benchmarks. Removing Nshared collapses performance to zero, while random ablations of the same size only moderately reduce scores. Higher is better for all metrics.
Figure 11: Word clouds of token concepts for different neurons.
Figure 12: Math neuron ratios across text and vision models. Vision models (red) and text models (blue) are compared by their proportion of math-related neurons, showing that a substantial fraction of shared neurons are activated by mathematical concepts.
Model Name
Params
Vision Encoder / Language Backbone
Merged / Text Backbone
Checkpoint
Idefics3‐8B‐LLaMA3
8B
SigLip vision + LLaMA3 (8B)
DeepSeek-LLaMA3
HuggingFaceM4/Idefics3-8B-LLaMA3
LLaVA-Next-8B
8B
(vision encoder as used in LLaVA-NeXT) + LLaMA3 (8B)
Although Large Vision Language Models (LVLMs) have demonstrated impressive multimodal reasoning capabilities, their scalability and deployment are constrained by massive computational requirements. In particular, the massive amount of visual tokens from high-resolution input data aggravates the situation due to the quadratic complexity of attention mechanisms. To address these issues, the research community has developed several optimization frameworks. This paper presents a comprehensive survey of the current state-of-the-art techniques for accelerating LVLM inference. We introduce a systematic taxonomy that categorizes existing optimization frameworks into four primary dimensions: visual token compression, memory management and serving, efficient architectural design, and advanced decoding strategies. Furthermore, we critically examine the limitations of these current methodologies and identify critical open problems to inspire future research directions in efficient multimodal systems.
Surendra Pathak, Bo Han
Computer Science Department, George Mason University, Fairfax, VA - 22030, USA
Large vision-language models (LVLMs) achieve impressive performance, yet their internal decision-making processes remain opaque, making it difficult to determine if the success stems from true multimodal fusion or from reliance on unimodal priors. To address this attribution gap, we introduce a novel framework using partial information decomposition (PID) to quantitatively measure the "information spectrum" of LVLMs -- decomposing a model's decision-relevant information into redundant, unique, and synergistic components. By adapting a scalable estimator to modern LVLM outputs, our model-agnostic pipeline profiles 26 LVLMs on four datasets across three dimensions -- breadth (cross-model & cross-task), depth (layer-wise information dynamics), and time (learning dynamics across training). Our analysis reveals two key results: (i) two task regimes (synergy-driven vs. knowledge-driven) and (ii) two stable, contrasting family-level strategies (fusion-centric vs. language-centric). We also uncover a consistent three-phase pattern in layer-wise processing and identify visual instruction tuning as the key stage where fusion is learned. Together, these contributions provide a quantitative lens beyond accuracy-only evaluation and offer insights for analyzing and designing the next generation of LVLMs. Code and data are available at https://github.com/RiiShin/pid-lvlm-analysis .
Recent work has increasingly explored neuron-level interpretation in vision-language models (VLMs) to identify neurons critical to final predictions. However, existing neuron analyses generally focus on single tasks, limiting the comparability of neuron importance across tasks. Moreover, ranking strategies tend to score neurons in isolation, overlooking how task-dependent information pathways shape the write-in effects of feed-forward network (FFN) neurons. This oversight can exacerbate neuron polysemanticity in multi-task settings, introducing noise into the identification and intervention of task-critical neurons. In this study, we propose HONES (Head-Oriented Neuron Explanation & Steering), a gradient-free framework for task-aware neuron attribution and steering in multi-task VLMs. HONES ranks FFN neurons by their causal write-in contributions conditioned on task-relevant attention heads, and further modulates salient neurons via lightweight scaling. Experiments on four diverse multimodal tasks and two popular VLMs show that HONES outperforms existing methods in identifying task-critical neurons and improves model performance after steering. Our source code is released at: https://github.com/petergit1/HONES.
Qidong Wang, Junjie Hu, Ming Jiang
Tongji University · University of Wisconsin-Madison