Current Multimodal Large Language Models (MLLMs) have advanced rapidly and follow two main paradigms: one involves concatenating visual and textual tokens for input into Large Language Model (LLM), while the other injects visual knowledge into Feed-Forward Networks (FFNs) of the LLM. However, the former introduces a large number of redundant visual tokens, substantially increasing computational overhead. The latter reduces sequence length but removes images from the input stage entirely, which may hinder the activation of image-related knowledge. Moreover, naively injecting visual knowledge into the FFNs could induce activation distribution shifts due to cross-modal semantic discrepancies. To address these challenges, we develop Beyond Token Stacking (BTS), a novel visual enhancement framework for MLLMs. Specifically, we propose a Multi-scale Visual Modulation (MVM) mechanism that injects multi-scale visual information at the input stage via modulation without increasing the token sequence length. Furthermore, we develop an Adaptive Fused-feature Calibration (AFC) module that performs per-channel affine calibration to decouple visual injection from distribution alignment in the FFN layers, improving multimodal fusion stability while preserving language priors. Experiments show that our approach achieves state-of-the-art performance.
Figures & tables
Figure 1. Framework comparisons between previous methods and our method.
Figure 2. Method overview. BTS consists of two principal components: a Multi-scale Visual Modulation (MVM) mechanism and an Adaptive Fused-feature Calibration (AFC) module. MVM injects multi-scale visual contextual information into text embeddings via lightweight residual modulation without increasing the token sequence length, whereas AFC calibrates cross-modal fused features by applying per-channel affine transformations in the FFN layers, thereby stabilizing cross-modal fusion.
Method
LLM
Subject
Context Modality
Grade
Average
NAT
SOC
LAN
TXT
IMG
NO
G1-6
G7-12
MLLM methods
Human ( Lu et al., 2022b )
-
90.23
84.97
87.48
89.60
87.50
88.10
91.59
82.42
88.40
GPT-3.5 ( Lu et al., 2022b )
GPT-3.5
74.64
69.74
76.00
74.44
67.28
77.42
76.80
68.89
73.97
GPT-3.5 w/ CoT ( Lu et al., 2022b )
GPT-3.5
75.44
70.87
78.09
74.68
67.43
79.93
78.23
69.68
75.17
GPT-4 ( Achiam et al., 2023 )
GPT-4
84.06
73.45
87.36
81.87
70.75
90.73
84.69
79.10
82.69
Table 1. Accuracy on ScienceQA test set. Question categories: NAT = natural science, SOC = social science, LAN = language science, TXT = w/ text context, IMG = w/ image context, NO = no context, G1-6 = grades 1-6, G7-12 = grades 7-12. Our results are shaded in gray. † Results from our reproduction of the original work. Other results are quoted from their original papers.
Method
GQA
VQAv2
OKVQA
VizWiz
SQA
MLLM methods
BLIP-2 ( Li et al., 2023 )
41.0
41.0
45.9
19.6
74.2
InstructBLIP-8B ( Dai et al., 2023 )
49.2
51.9
49.9
34.5
73.3
IDEFICS ( Laurençon et al., 2023 )
38.4
50.9
38.4
35.5
77.4
LLaVA ( Liu et al., 2023 )
49.5
63.6
45.2
30.7
56.5
LLaVA-v1.5 ( Liu et al., 2024 )
62.0
74.0
45.6
50.0
90.9
Table 2. Comparative performance evaluation on multiple VQA benchmarks. Best result is in bold face.
#
MVM
AFC
OKVQA
VizWiz
SQA
down
up
0
✓
-
-
52.1
70.8
93.2
1
-
✓
✓
51.1
70.1
92.9
2
✓
✓
-
51.5
71.9
93.3
3
✓
-
✓
51.3
71.3
92.8
4
✓
✓
✓
52.5
72.5
93.6
Table 3. Ablation study on each component. down and up represent two projections of FFN, respectively.
Figure 3. Case study comparing our method with other methods on challenging visual reasoning tasks.
MVM
OKVQA
VizWiz
SQA
[256]
51.5
71.8
93.2
[256,64]
52.5
72.5
93.6
[256,64,16]
51.8
72.0
93.1
Table 4. Ablation study on Multi-scale Visual Modulation (MVM) scale settings.
Method
LLM backbone
SQA
LLaMA-7B
Vicuna-7B
ADEM-VL
✓
-
92.88
MemVP
✓
-
93.07
Ours
✓
-
93.59
ADEM-VL
-
✓
91.75
MemVP
-
✓
91.26
Table 5. Ablation study on different LLM backbones on SQA.
Figure 4. Visualization of attention on image patches.
Figure 5. Visualization of the distributions of per-channel activation standard deviation before (pre) and after (post) AFC.
Figure 6. Average training time (s/iter) across tasks.
Multimodal large language models (MLLMs) commonly inherit the deep, symmetric Transformer backbone designed for unimodal text modeling, and apply the same computation uniformly to image and language tokens. This design overlooks a key modality asymmetry: image and text tokens differ substantially in information density, redundancy, and required reasoning depth. Through a layer-wise analysis of LLaVA-1.5, we observe that vision tokens tend to saturate in the middle layers. Specifically, text-to-image attention decreases from 0.68 at layer 0 to 0.07 by layer 4, and stabilizes near 0.04 after layer 18, whereas text tokens continue to benefit from deep semantic processing. These findings suggest a mismatch between architectural symmetry and depth-asynchronous modality evolution, resulting in redundant visual computation and possible drift in perceptual representations during deep task-specific adaptation. Motivated by this, we propose Dual-Path Vision Token Routing (DPVR), a modality-asymmetric routing framework for efficient MLLMs. Its core instantiation, DPVR-LF (Late-Layer Fusion), routes vision tokens at the saturation point into a one-layer trainable side branch, runs a thirteen-layer text-only forward that skips image positions in the deep stack, and re-fuses the visual and textual streams only at the final layer. With approximately 3% trainable parameters, DPVR-LF preserves competitive multimodal performance on standard benchmarks while reducing visual computation in the deep Transformer stack. The results challenge the conventional assumption that vision tokens must traverse all deep language-model layers, and indicate that a single late fusion layer can be sufficient for maintaining strong perceptual competence in LLaVA-style MLLMs.
Siyuan Liu, Jinyang Wu
aSchool of Mechanics and Engineering Science, Peking University, China · bDepartment of Automation, Tsinghua University, China
Multimodal Large Language Models (MLLMs) suffer from substantial computational overhead due to the high redundancy in visual token sequences. Existing approaches typically address this issue using single-layer Vision Transformer (ViT) features and static pruning strategies. However, such fixed configurations are often brittle under diverse instructions. To overcome these limitations, we propose CLASP, a plug-and-play token reduction framework based on class-adaptive layer fusion and dual-stage pruning. Specifically, CLASP first constructs category-specific visual representations through multi-layer vision feature fusion. It then performs dual-stage pruning, allocating the token budget between attention-salient pivot tokens for relevance and redundancy-aware completion tokens for coverage. Through class-adaptive pruning, CLASP enables prompt-conditioned feature fusion and budget allocation, allowing aggressive yet robust visual token reduction. Extensive experiments demonstrate that CLASP consistently outperforms existing methods across a wide range of benchmarks, pruning ratios, and MLLM architectures. Code will be available at https://github.com/Yunkaidang/CLASP.
Yunkai Dang, Yizhu Jiang, Yifan Jiang +4
School of Artificial Intelligence Science and Technology, Nanjing University.
Multimodal Large Language Models (MLLMs) have achieved remarkable performance by aligning pretrained visual representations with the linguistic knowledge embedded in Large Language Models (LLMs). However, existing approaches typically rely on final-layer visual features or learnable multi-layer fusion, which often fail to sufficiently exploit hierarchical visual cues without explicit cross-layer interaction design. In this work, we propose a Memory-Augmented Adapter (Mema) within the vision encoder. Specifically, Mema maintains a stateful memory that accumulates hierarchical visual representations across layers, with its evolution conditioned on both query embeddings and step-wise visual features. A portion of this memory is selectively injected into token representations via a feedback mechanism, thereby mitigating the attenuation of fine-grained visual cues from shallow layers. Designed as a lightweight and plug-and-play module, Mema integrates seamlessly into pretrained vision encoders without modifying the vanilla backbone architecture. Only a minimal set of additional parameters requires training, enabling adaptive visual feature refinement while reducing training overhead. Extensive experiments across multiple benchmarks demonstrate that Mema consistently improves performance, validating its effectiveness in complex multimodal reasoning tasks. The code have been released at https://github.com/Sisiliu312/Mema.
Ying Liu, Yudong Han, Kean Shi +1
Beijing Institute of Technology · Peking University · Yangtze Delta Region Academy of Beijing Institude of Technology, Jiaxing, China