iGVLM: Dynamic Instruction-Guided Vision Encoding for Question-Aware Multimodal Understanding
Authors: Hanpeng Liu, Yaqian Li, Zidan Wang, Shuoxi Zhang, Zihao Bo, Rinyoichi Takezoe, Kaiwen Long, Kun He
Organizations: School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China · Li Auto Inc. · Institute of AI for Industries, Chinese Academy of Sciences.
Despite the success of Large Vision--Language Models (LVLMs), most existing architectures suffer from a representation bottleneck: they rely on static, instruction-agnostic vision encoders whose visual representations are utilized in an invariant manner across different textual tasks. This rigidity hinders fine-grained reasoning where task-specific visual cues are critical. To address this issue, we propose iGVLM, a general framework for instruction-guided visual modulation. iGVLM introduces a decoupled dual-branch architecture: a frozen representation branch that preserves task-agnostic visual representations learned during pre-training, and a dynamic conditioning branch that performs affine feature modulation via Adaptive Layer Normalization (AdaLN). This design enables a smooth transition from general-purpose perception to instruction-aware reasoning while maintaining the structural integrity and stability of pre-trained visual priors. Beyond standard benchmarks, we introduce MM4, a controlled diagnostic probe for quantifying logical consistency under multi-query, multi-instruction settings. Extensive results show that iGVLM consistently enhances instruction sensitivity across diverse language backbones, offering a plug-and-play paradigm for bridging passive perception and active reasoning.
Figures & tables
Figure 1: Visualization of vision features. We employ Grad-CAM to visualize the vision encoders of the two branches in iGVLM, highlighting the regions most relevant to the correct answer. As shown in the figure, the instruction-guided branch distinctly focuses on areas that are more closely associated with the correct answer.
Figure 2: (a): The proposed iGVLM architecture. The Text Encoder extracts features from the input instructions to guide the Vision Encoder, enabling dynamic modulation of visual representations. These instruction-conditioned features are then fused with static visual features. The fused representation is aligned with a Large Language Model (LLM) to generate responses. The illustrated example comes from a real-world VQA scenario rather than the MM4 benchmark. (b): AdaLN-Modified ViT. We leverage textual information to modulate the multi-head attention and MLP modules within the ViT the AdaLN adapter, enabling instruction-aware adjustment of visual attention.
Method
Venue
CP
FP
IR
LR
ST
MA
Avg.
Δ
Thpt
Vicuna-7B Backbone
LLaVA-1.5
CVPR’24
58.8
24.0
38.8
24.0
13.6
22.8
30.3
–
13.5
QA-ViT
CVPR’24
60.0
28.0
40.4
25.6
14.0
21.2
31.5
+1.2
12.9
DyFo
CVPR’25
54.8
28.4
38.0
25.6
24.0
27.2
33.0
+2.7
0.49
iGVLM
Ours
59.6
28.0
41.2
32.8
19.6
26.8
34.7
+4.4
11.1
Vicuna-13B Backbone
Table 1: Comparison among dynamic visual perception methods on MMStar , evaluated across six perception dimensions (CP, FP, IR, LR, ST, MA), average accuracy (Avg.), and throughput (it/s). Our iGVLM consistently outperforms prior approaches under both Vicuna and Qwen2.5 backbones, achieving the best overall balance between accuracy and efficiency. Δ indicates improvement over LLaVA-1.5 under the same backbone. Bold and underlined values indicate the best and second-best results, respectively.
Figure 3: Representative Examples from MM4. Deceptively simple questions demand multiperspective reasoning. Correct answers (highlighted) are uniformly distributed across options to prevent positional bias. More examples can be found in Appendix C .
n
GPT-4o
Qwen2.5-vl-max
GPT-4v
iGVLM-13B
QA-ViT-13B
LLaVA1.5-13B
iGVLM-7B
LLaVA1.5-7B
QA-ViT-7B
LLaVA-1.5-3B
iGVLM-3B
1
170
172
161
161
163
161
161
157
162
165
164
2
138
149
120
117
115
113
109
113
108
130
124
3
107
114
66
71
68
77
58
61
59
82
78
4
58
52
24
23
21
18
17
13
11
27
29
Table 2: Scores of different methods on MM4. Different values of n indicate different scoring schemes.
Method
LLM
VQAv2
GQA
POPE
VisWiz
SQA
LLaVA-1.5
Vicuna-7B
78.5
62.0
85.4
50.0
66.8
QA-ViT
Vicuna-7B
79.0
62.2
85.7
51.3
68.1
DyFo
Vicuna-7B
-
-
84.8
-
65.5
iGVLM
Vicuna-7B
79.1
62.8
85.9
52.5
69.9
LLaVA-1.5
Vicuna-13B
80.0
63.3
85.4
53.6
71.6
QA-ViT
Vicuna-13B
79.9
63.2
85.8
56.0
71.3
Table 3: Comparison with existing VLMs on other benchmarks including VQAv2, GQA, POPE, VisWiz, and SQA. Our iGVLM consistently achieves higher accuracy across all backbones, confirming its strong generalization beyond MMStar and MM4.
Method
Params(B)
MM4
MMStar
VQAv2
Viswiz
LLaVA1.5-13B
13.35
19
32.8
80.0
53.6
iGVLM-13B
13.78
23
36.4
80.2
55.3
-w/o AdaLN
13.78
22
35.1
80.2
53.5
-w/o FFN
13.78
17
34.1
80.1
54.7
-w/o Pure
13.48
5
27.3
60.2
37.9
Table 4: Component ablation of iGVLM . We analyze the effect of removing key modules including AdaLN, Zero-FFN, and the static branch. Performance consistently drops, confirming that instruction-guided modulation, Zero-FFN fusion, and dual-branch structure are all critical for iGVLM’s effectiveness.
Method
MMStar
VQAv2
GQA
VisWiz
POPE
iGVLM
34.7
79.1
62.8
69.9
85.9
iGVLM-MoF
32.0
79.3
62.7
68.8
85.4
iGVLM-Cross
33.0
79.4
62.8
68.6
85.8
Table 5: Comparison of iGVLM variants on multiple benchmarks. iGVLM-MoF and iGVLM-Cross replace AdaLN with interleaved or cross-attention modulation, but both yield lower accuracy, highlighting the efficiency of our AdaLN-based design.
Model
LLM
MMStar (Avg.)
MM4 (Score)
LLaVA-1.5B
Qwen2.5-1.5B
17.1
19
iGVLM-1.5B
Qwen2.5-1.5B
19.7
16
iGVLM-3B
Qwen2.5-3B
21.3
29
iGVLM-7B
Vicuna-7B
34.7
17
iGVLM-13B
Vicuna-13B
36.4
23
Table 6: Scaling performance of iGVLM on MMStar and MM4 benchmarks under different model sizes and LLM backbones.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Pretrain
Finetune
Batch size
256
128
Lr
6e-4
2e-5
Lr schedule
cosine decay
Lr warmup ratio
0.03
Weight decay
0
Epoch
1
Appendix
Table 7: Hyperparameters of iGVLM are the same as the original LLaVA-1.5, except the learning rate in pretraining.