Current Multimodal Large Language Models (MLLMs) have advanced rapidly and follow two main paradigms: one involves concatenating visual and textual tokens for input into Large Language Model (LLM), while the other injects visual knowledge into Feed-Forward Networks (FFNs) of the LLM. However, the former introduces a large number of redundant visual tokens, substantially increasing computational overhead. The latter reduces sequence length but removes images from the input stage entirely, which may hinder the activation of image-related knowledge. Moreover, naively injecting visual knowledge into the FFNs could induce activation distribution shifts due to cross-modal semantic discrepancies. To address these challenges, we develop Beyond Token Stacking (BTS), a novel visual enhancement framework for MLLMs. Specifically, we propose a Multi-scale Visual Modulation (MVM) mechanism that injects multi-scale visual information at the input stage via modulation without increasing the token sequence length. Furthermore, we develop an Adaptive Fused-feature Calibration (AFC) module that performs per-channel affine calibration to decouple visual injection from distribution alignment in the FFN layers, improving multimodal fusion stability while preserving language priors. Experiments show that our approach achieves state-of-the-art performance.
Figures & tables
Figure 1. Framework comparisons between previous methods and our method.
Figure 2. Method overview. BTS consists of two principal components: a Multi-scale Visual Modulation (MVM) mechanism and an Adaptive Fused-feature Calibration (AFC) module. MVM injects multi-scale visual contextual information into text embeddings via lightweight residual modulation without increasing the token sequence length, whereas AFC calibrates cross-modal fused features by applying per-channel affine transformations in the FFN layers, thereby stabilizing cross-modal fusion.
Method
LLM
Subject
Context Modality
Grade
Average
NAT
SOC
LAN
TXT
IMG
NO
G1-6
G7-12
MLLM methods
Human ( Lu et al., 2022b )
-
90.23
84.97
87.48
89.60
87.50
88.10
91.59
82.42
88.40
GPT-3.5 ( Lu et al., 2022b )
GPT-3.5
74.64
69.74
76.00
74.44
67.28
77.42
76.80
68.89
73.97
GPT-3.5 w/ CoT ( Lu et al., 2022b )
GPT-3.5
75.44
70.87
78.09
74.68
67.43
79.93
78.23
69.68
75.17
GPT-4 ( Achiam et al., 2023 )
GPT-4
84.06
73.45
87.36
81.87
70.75
90.73
84.69
79.10
82.69
Table 1. Accuracy on ScienceQA test set. Question categories: NAT = natural science, SOC = social science, LAN = language science, TXT = w/ text context, IMG = w/ image context, NO = no context, G1-6 = grades 1-6, G7-12 = grades 7-12. Our results are shaded in gray. † Results from our reproduction of the original work. Other results are quoted from their original papers.
Method
GQA
VQAv2
OKVQA
VizWiz
SQA
MLLM methods
BLIP-2 ( Li et al., 2023 )
41.0
41.0
45.9
19.6
74.2
InstructBLIP-8B ( Dai et al., 2023 )
49.2
51.9
49.9
34.5
73.3
IDEFICS ( Laurençon et al., 2023 )
38.4
50.9
38.4
35.5
77.4
LLaVA ( Liu et al., 2023 )
49.5
63.6
45.2
30.7
56.5
LLaVA-v1.5 ( Liu et al., 2024 )
62.0
74.0
45.6
50.0
90.9
Table 2. Comparative performance evaluation on multiple VQA benchmarks. Best result is in bold face.
#
MVM
AFC
OKVQA
VizWiz
SQA
down
up
0
✓
-
-
52.1
70.8
93.2
1
-
✓
✓
51.1
70.1
92.9
2
✓
✓
-
51.5
71.9
93.3
3
✓
-
✓
51.3
71.3
92.8
4
✓
✓
✓
52.5
72.5
93.6
Table 3. Ablation study on each component. down and up represent two projections of FFN, respectively.
Figure 3. Case study comparing our method with other methods on challenging visual reasoning tasks.
MVM
OKVQA
VizWiz
SQA
[256]
51.5
71.8
93.2
[256,64]
52.5
72.5
93.6
[256,64,16]
51.8
72.0
93.1
Table 4. Ablation study on Multi-scale Visual Modulation (MVM) scale settings.
Method
LLM backbone
SQA
LLaMA-7B
Vicuna-7B
ADEM-VL
✓
-
92.88
MemVP
✓
-
93.07
Ours
✓
-
93.59
ADEM-VL
-
✓
91.75
MemVP
-
✓
91.26
Table 5. Ablation study on different LLM backbones on SQA.
Figure 4. Visualization of attention on image patches.
Figure 5. Visualization of the distributions of per-channel activation standard deviation before (pre) and after (post) AFC.
Figure 6. Average training time (s/iter) across tasks.