Most Vision-Language Models (VLMs) are built by extending pretrained Large Language Models (LLMs) with visual modules and multimodal alignment. However, this multimodal scaling often degrades the language-side reasoning ability originally encoded in the base LLM. While the base LLM retains usable reasoning after scaling, the aligned VLM itself cannot reliably access this ability. Therefore, recovering the degraded reasoning capability in VLMs would benefit more from seeking help from the base LLM than from the VLM alone. Motivated by this, we propose LIFT (Language-side reasonIng Facilitation and Transfer), a lightweight vector-intervention method that transfers reasoning capability from the base LLM to the VLM without retraining the backbone. LIFT defines Reasoning Vectors as answer-token hidden-state differences between a Reasoner path with an explicit reasoning trace and a Solver path without it, and injects these vectors into language-side activations of the target VLM. LIFT further supports learnable vector adaptation while keeping the VLM backbone frozen. We evaluate LIFT on two VLMs across six reasoning benchmarks, comparing Reasoning Vectors extracted from the base LLM and from the aligned VLM under matched protocols. Results show that LLM-derived vectors consistently outperform VLM-derived vectors, confirming that the base LLM is a more effective source for recovering reasoning. LIFT partially recovers degraded reasoning through lightweight language-side interventions. Further analyses show that Reasoning Vectors influence intermediate reasoning behavior rather than merely altering final answers. The source code will be released soon.
Figures & tables
Figure 1: Motivation of LIFT. Multimodal alignment may change language-side reasoning directions in the VLM. LIFT tests whether Reasoning Vectors from the base LLM can be reused as lightweight interventions in the aligned VLM.
Figure 2: Overview of LIFT. (a) For each layer, LIFT computes hidden-state differences between the Reasoner and Solver paths and averages these differences to obtain the Reasoning Vector. (b) Learnable adaptation optimizes only the injected vector parameters while keeping the VLM backbone frozen. (c) At inference, vectors are injected into selected VLM layers.
Model
Method
GSM8K
MVista
MVision
CSQA
SQA
SciQA
Avg.
Qwen2.5-VL
Baseline
82.2
68.5
25.0
77.3
58.6
87.8
66.6
1-shot CoT
82.7
68.4
25.3
75.1
61.2
87.9
66.8
2-shot CoT
82.9
68.4
25.3
76.8
61.3
87.9
67.1
4-shot CoT
82.9
68.4
25.3
78.1
61.6
87.9
67.4
LLM-derived
84.2
69.1
26.6
78.0
63.5
88.1
68.3
VLM-derived
83.0
70.7
25.7
77.7
60.3
87.8
67.5
Table 1: Main results on six reasoning benchmarks across two VLMs. We abbreviate MathVista/MathVision/CommonsenseQA/StrategyQA/ScienceQA as MVista/MVision/CSQA/SQA/SciQA. LLM/VLM-derived denote static vector transfer using Reasoning Vectors extracted from the base LLM and the corresponding VLM, respectively; LLM/VLM-learnable are learnable vectors initialized from these corresponding static vectors. Bold and underlined numbers indicate the best and second-best results for each model and benchmark. All results are reported as accuracy (%), and Avg. denotes the average accuracy across benchmarks.
Figure 3: Representative Case Studies of Reasoning Repairs.
Model
Method
MVision
SciQA
Avg.
Qwen2.5-VL
Baseline
25.0
87.8
56.4
LLM-derived
26.6
88.1
57.4
VLM-derived
25.7
87.8
56.8
VLM-MM
26.0
87.9
57.0
InternVL2.5
Baseline
22.7
96.5
59.6
LLM-derived
25.0
96.8
60.9
Table 2: Accuracy (%) of VLM multimodal extraction on multimodal reasoning benchmarks.
Model
Method
GSM8K
MVista
MVision
CSQA
SQA
SciQA
Avg.
Qwen2.5-VL
LLM-learnable
85.6
71.8
27.3
78.3
62.4
88.9
69.1
Random init
83.2
68.6
24.7
77.4
58.8
87.9
66.7
VLM-learnable
83.2
69.5
24.7
77.7
60.3
87.9
67.2
Random init
83.0
68.6
24.7
77.4
58.8
87.9
66.7
InternVL2.5
LLM-learnable
78.0
64.6
25.3
83.2
67.6
97.4
69.4
Random init
77.4
63.3
24.1
83.1
65.4
96.7
68.3
Table 3: Initialization ablation for learnable vector adaptation. Random init replaces the extracted Reasoning Vector with a random vector under the same training setting.
Figure 4: Cross-layer cosine similarity between LLM- and VLM-derived vectors on GSM8K.
Figure 5: Layer-wise performance of LLM- and VLM-derived vectors averaged over six datasets.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Modality
#Eval. examples
Support source
GSM8K
Text
1319
GSM8K
MathVista
Image-text
1000
GSM8K
MathVision
Image-text
304
GSM8K
CommonsenseQA
Text
1221
CommonsenseQA
StrategyQA
Text
687
StrategyQA
ScienceQA
Image-text
4241
CommonsenseQA
Appendix
Table 4: Evaluation split sizes and support sources used in our experiments. Support examples are used only for offline Reasoning Vector extraction or learnable adaptation and are never appended to evaluation prompts.
Setting
Prompt Template
Direct Answer
Please answer the question directly. Put your final answer within \boxed{}.
Reasoning Trace
Please reason step by step. Put your final answer within \boxed{}.
Multiple Choice
Please choose the correct option. Put your final answer within \boxed{}.
Yes/No
Please answer the question with yes or no. Put your final answer within \boxed{}.
Appendix
Table 5: Prompt templates used in our experiments.
Parameter
Value
Number of beams
1
Maximum new tokens
512 / 4096
Length penalty
0.0
Random seed
42
Appendix
Table 6: Generation configuration used for evaluation.
Hyperparameter
Value
Trainable parameters
Injected vectors θl only
Backbone parameters
Frozen
Optimizer
AdamW
Learning rate
1×10−4
Batch size
1
Gradient accumulation
4
Appendix
Table 7: Hyperparameters for learnable vector adaptation.
Benchmark
#Examples
GSM8K
1319
MathVista
1000
MathVision
304
CommonsenseQA
1221
StrategyQA
687
ScienceQA
4241
Appendix
Table 8: Evaluation split sizes used in our experiments.
Comparison
Avg. Delta
95% CI
LLM-derived − Baseline
+1.45
[0.45, 2.45]
VLM-derived − Baseline
+0.70
[0.12, 1.28]
LLM-derived − VLM-derived
+0.75
[0.15, 1.35]
Appendix
Table 9: Paired bootstrap confidence intervals for macro-average accuracy differences.
Model
Method
GSM8K
MathVista
MathVision
CSQA
SQA
SciQA
Avg.
Qwen2.5-VL
Baseline
82.2
68.5
25.0
77.3
58.6
87.8
66.6
LLM-derived
84.4
69.5
26.7
78.3
63.4
88.0
68.4
VLM-derived
83.2
70.6
25.7
77.8
60.3
87.6
67.5
InternVL2.5
Baseline
77.1
62.8
22.7
82.3
65.6
96.5
67.8
LLM-derived
78.5
63.9
25.0
83.5
67.2
96.8
69.2
VLM-derived
77.5
63.1
24.3
82.9
65.8
96.7
68.4
Appendix
Table 10: Additional-seed robustness check for frozen Reasoning Vector transfer.
Model
Method
GSM8K
MathVista
MathVision
CSQA
SQA
SciQA
Avg.
Qwen2.5-VL
Baseline
82.2
68.5
25.0
77.3
58.6
87.8
66.6
Same-norm random
82.1
68.2
25.0
77.3
58.2
87.8
66.4
LLM-derived
84.2
69.1
26.6
78.0
63.5
88.1
68.3
InternVL2.5
Baseline
77.1
62.8
22.7
82.3
65.6
96.5
67.8
Same-norm random
77.1
62.6
22.8
79.7
58.4
96.3
66.1
LLM-derived
78.1
64.0
25.0
83.1
67.1
96.8
69.0
Appendix
Table 11: Same-norm random-vector control. All results are reported as accuracy (%).
Model
Method
GSM8K
MVista
MVision
CSQA
SQA
SciQA
Avg.
Qwen2.5-VL
Baseline
82.2
68.5
25.0
77.3
58.6
87.8
66.6
LLM-derived
83.2
69.6
26.6
77.8
59.7
88.1
67.5
VLM-derived
82.9
69.5
26.3
77.6
59.4
88.0
67.3
InternVL2.5
Baseline
77.1
62.8
22.7
82.3
65.6
96.5
67.8
LLM-derived
78.1
63.9
25.3
83.2
67.4
96.7
69.1
VLM-derived
77.9
63.2
25.0
83.1
67.0
96.6
68.8
Appendix
Table 12: Additional results of Attention-level Injection. For each dataset, the injection layer is selected on a held-out development subset and then fixed for evaluation.
College of Computing and Data Science, Nanyang Technological University, Singapore · Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · Tongyi Lab, Alibaba Group, China