A central yet under-explored question in Multimodal Large Language Models (MLLMs) is where to connect---how visual information from a hierarchical vision encoder should be wired into the layer-wise semantics of a large language model. Mainstream MLLMs inject a fixed single-layer visual representation into the LLM and treat all LLM layers as a single consumer, ignoring that different layers demand different visual granularities. Recent multi-layer fusion methods either compress hierarchical features on the encoder side or rely on predefined sparse or hierarchical layer-to-layer connections, and therefore stop short of modeling adaptive cross-layer interactions between the ViT and LLM hierarchies. We address this gap with DGP (Dynamic Gated Pathways), a routing mechanism that establishes all-to-all connections between every ViT layer and every LLM layer, with each LLM layer dynamically gating its preference over visual layers conditioned on its own semantic state. Built on top of LLaVA-1.5-7B and 13B, DGP delivers consistent gains over the LLaVA-1.5-7B baseline (e.g., +4.2 on SQAI, +3.3 on MM-Vet, +3.1 on VizWiz and LLaVAW) and surpasses prior multi-layer visual fusion methods on most benchmarks. Layer-wise routing analyses further show that different LLM layers do select different visual granularities on demand, supporting both the effectiveness and interpretability of the proposed dynamic gated pathways. The code is available: https://github.com/YingyingYan1/DGP.
Figures & tables
Figure 1: Attention distributions across ViT layers. Shallow layers show dispersed attention over edges and textures; deeper layers progressively concentrate on a few semantic anchors.
Figure 2: Strategies for visual token insertion. (A) Fixed single-layer connection. (B) Encoder-side fusion of multiple ViT layers into one representation. (C) Predefined sparse or hierarchical connections between ViT and LLM layers. (D) DGP (Ours): dynamic gated pathways between all ViT and all LLM layers. wm denotes the routing weights that LLM layer m assigns across all ViT layers.
Figure 3: Overview of DGP . (A) At every LLM layer, the Dynamic Gated Pathways module routes over all ViT layers conditioned on the current language-layer state, aggregates the selected visual features, and injects them back into the LLM hidden states through a gated residual connection—establishing all-to-all cross-layer connections between ViT and LLM. (B) Inside the module, ViT layer summaries serve as keys/values and the language-layer state as the query; a shared cross-attention followed by a layer-specific routing MLP produces the per-layer routing weights. Weight Matrix (m) denotes the routing weights of LLM layer m over all ViT layers.
Academic-oriented VQA
LMM benchmarks
Method
LLM
GQA
SQA I
VQA v2
VizWiz
SEED
SEED I
LLaVA W
MM-Vet
MMB
MMB CN
POPE
BLIP-2 [ 19 ]
Vicuna-13B
41.0
61.0
65.0
19.6
46.4
49.7
38.1
22.4
–
–
85.3
InstructBLIP [ 11 ]
Vicuna-7B
49.2
60.5
–
34.5
53.4
58.8
60.9
26.2
36.0
23.7
–
InstructBLIP [ 11 ]
Vicuna-13B
49.5
63.1
–
33.4
78.9
–
58.2
25.6
–
–
78.9
Shikra [ 9 ]
Vicuna-13B
–
–
77.4
–
–
–
–
–
58.8
–
–
IDEFICS-9B [ 17 ]
LLaMA-7B
38.4
–
50.9
35.5
–
44.5
–
–
48.2
25.2
–
Table 1: Comparison with state-of-the-art MLLMs. We evaluate on Academic-oriented VQA tasks and recent LMM benchmarks. Best results are in bold ; second-best are underlined . “ -- ” denotes results not reported in the original papers. Rows below DGP report the absolute point gain ( ↑ ) or loss ( ↓ ) relative to LLaVA-1.5.
Figure 4: Qualitative comparison between DGP and LLaVA-1.5. Representative examples from math solving, OCR understanding, and detailed visual reasoning. LLaVA-1.5 often misses critical visual evidence or hallucinates content; DGP produces more accurate, evidence-grounded responses by adaptively accessing multi-granularity visual features.
Avg.
Gated
Entropy
Academic-oriented VQA
LMM benchmarks
Fusion
Routing
Reg.
GQA
SQA I
VQA v2
VizWiz
SEED
SEED I
LLaVA W
MM-Vet
MMB
MMB CN
POPE
✗
✗
✗
62.0
66.8
78.5
50.0
58.6
66.1
63.4
30.5
64.3
58.3
85.9
✓
✗
✗
60.2
64.3
72.0
50.2
57.9
63.3
56.3
25.3
57.6
44.9
77.2
✗
✓
✗
64.1
70.5
78.9
52.0
60.9
66.7
63.6
32.4
65.6
59.1
86.4
✗
✓
✓
64.6
71.0
79.7
53.1
61.5
67.5
66.5
33.8
66.7
60.5
87.0
Table 4: Ablation of DGP components. We evaluate the contribution of average fusion, gated routing, and entropy regularization.
Figure 5: Gated Pathway Analysis. (A) Heatmap of routing weights over LLM (x) and ViT (y) layers. (B) Routing curves of representative LLM layers with multimodal patterns. (C) Routing weights of LLM layer 21 across benchmarks, showing task-specific preferences. (D) Per-sample routing distributions within the same benchmark, showing sample-level diversity.
Figure 6: Task-aware routing for the same image. As the question changes, routing shifts: attribute recognition activates shallower layers, while global-semantic questions activate deeper ones.
Figure 7: Fused visual feature visualization. We compare CLIP layer 23, the uniform average over all 24 ViT layers, and DGP ’s gated fusion at different LLM layers.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Operation
Tensor Shape
Description
Vision Encoding (ViT)
Raw patches
ViT layer ℓ output
[B,577,1024]
1 CLS + 576 patches per layer
Patch features
Remove CLS
[B,576,1024]×24
Cache all 24 layers for routing
CLS tokens
Extract CLS
[B,1,1024]×24
Layer summaries for cross-attention
CLS Projection for Routing
CLS projection
LN(1024)→Linear(1024→4096)
[B,1,4096]×24
Project each CLS to LLM space
Appendix
Table 5: Token flow and tensor dimensions in DGP . We illustrate each processing stage using Vicuna-7B (32 layers, d=4096 ) and CLIP-ViT-L/14-336 (24 layers, dv=1024 , Nv=576 patches). B denotes batch size, Nt denotes the number of text tokens.
Pre-training (PT)
Supervised Fine-tuning (SFT)
Global batch size
256
Global batch size
128
Learning rate
1×10−3
Learning rate
2×10−5
Vision encoder LR
frozen
Vision encoder LR
frozen
LR schedule
cosine decay
LR schedule
cosine decay
LR warmup ratio
0.03
LR warmup ratio
0.03
Epochs
1
Epochs
1
Appendix
Table 6: Training hyperparameters. Configuration for pre-training (PT) on LLaVA-558K and supervised fine-tuning (SFT) on LLaVA-mixed-665K.
Category
Benchmark
Task Type
Evaluated Capabilities
Academic VQA
VQA v2 [ 13 ]
General VQA
Basic image understanding, object attribute recognition, and commonsense visual reasoning
GQA [ 15 ]
Compositional VQA
Compositional visual reasoning with structured scene-graph annotations (relations, attributes, and logical consistency)
VizWiz [ 14 ]
Real-world VQA
Real user questions from visually impaired users with often noisy images, testing robust visual understanding
SQA I [ 29 ]
Science QA (image)
Scientific knowledge, image-grounded reasoning, and multi-step inference
LMM Bench
POPE [ 22 ]
Hallucination probing
Contrastive image–question pairs to test visual grounding and robustness against object hallucination
MMBench (MMB/MMB CN ) [ 28 ]
Multi-choice
Classification-style questions covering perception, reasoning, and instruction following (English/Chinese subsets)
Appendix
Table 7: Benchmark descriptions. We evaluate on two complementary suites: academic task-oriented benchmarks and instruction-following / LMM benchmarks.
γinit
Academic-oriented VQA
LMM benchmarks
GQA
SQA I
VQA v2
VizWiz
SEED
SEED I
LLaVA W
MM-Vet
MMB
MMB CN
POPE
0.1
64.4
70.6
79.3
52.7
61.4
67.3
66.2
33.5
66.2
60.2
87.2
0.3 (default)
64.6
71.0
79.7
53.1
61.5
67.5
66.5
33.8
66.7
60.5
87.0
0.5
64.3
70.8
79.1
53.0
61.4
67.1
66.3
34.0
66.5
60.4
86.9
Appendix
Table 8: Effect of gating initialization γinit . We vary the initial value of the residual gate and report performance across the full benchmark suite.
Component
7B (M)
% of DGP
13B (M)
% of DGP
Routers (per-layer MLP)
135.3
37.80%
211.2
37.88%
Shared Cross-Attention
67.1
18.75%
104.9
18.81%
CLS Expert
4.2
1.17%
5.3
0.95%
Post-Fusion Experts
134.4
37.55%
210.0
37.66%
Others (embeddings, gates, norms)
16.9
4.72%
26.2
4.70%
Total DGP Module
357.9
100%
557.6
100%
Appendix
Table 9: Parameter breakdown of the DGP module.
Academic-oriented VQA
LMM benchmarks
Method
LLM
GQA
SQA I
VQA v2
VizWiz
SEED
SEED I
LLaVA W
MM-Vet
MMB
MMB CN
POPE
LLaVA-1.5 [ 25 ]
Vicuna-7B
62.0
66.8
78.5
50.0
58.6
66.1
63.4
30.5
64.3
58.3
85.9
DGP
Vicuna-7B
64.6
71.0
79.7
53.1
61.5
67.5
66.5
33.8
66.7
60.5
87.0
DGP ‡
Vicuna-7B
64.1
71.0
78.9
50.2
59.7
65.3
63.5
33.7
68.3
59.4
84.8
Appendix
Table 10: Effect of unfreezing the vision encoder. Comparison between DGP with frozen and unfrozen ( ‡ ) vision encoder during training.
LLM
ViT Layer
Layer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
1
2.4
2.1
1.9
2.9
16.8
2.2
38.3
2.7
5.8
6.8
2.8
2.2
1.8
1.5
1.6
1.4
1.3
1.1
0.9
0.8
0.7
0.6
0.6
0.6
2
2.0
2.0
1.8
2.6
2.3
2.3
2.7
4.6
4.8
5.1
6.3
4.0
19.1
12.1
10.2
9.5
2.8
1.6
1.0
0.8
0.7
0.6
0.5
0.6
3
1.7
1.5
1.3
1.3
1.4
1.3
1.5
1.5
1.8
1.9
2.1
11.0
58.8
2.5
2.5
1.7
1.4
1.1
1.0
0.7
0.6
0.5
0.4
0.5
4
1.8
1.8
1.7
1.9
1.9
2.0
3.0
3.5
4.3
2.9
3.4
11.6
44.9
2.6
3.3
2.7
1.7
1.2
1.0
0.8
0.6
0.5
0.4
0.5
5
1.7
1.7
1.6
1.6
1.5
1.6
1.8
2.0
3.2
2.6
4.1
20.8
11.3
29.9
3.8
3.2
1.9
1.5
1.0
0.8
0.7
0.6
0.5
0.7
Appendix
Table 11: Full routing weight matrix on POPE. Each cell shows the percentage weight ( ×100 ) that LLM layer l assigns to ViT layer v . Rows sum to 100.
Routing
Academic-oriented VQA
LMM benchmarks
GQA
SQA I
VQA v2
VizWiz
SEED
SEED I
LLaVA W
MM-Vet
MMB
MMB CN
POPE
DGP (Learned)
64.6
71.0
79.7
53.1
61.5
67.5
66.5
33.8
66.7
60.5
87.0
Baseline (Last-2)
62.0
66.8
78.5
50.0
58.6
66.1
63.4
30.5
64.3
58.3
85.9
Uniform
60.2
64.3
72.0
50.2
57.9
63.3
56.3
25.3
57.6
44.9
77.2
Reversed
63.6
67.5
77.8
52.4
61.3
67.1
57.0
32.1
61.7
55.8
86.6
Appendix
Table 12: Inference-time routing intervention. Comparison of different routing strategies on DGP-7B without retraining.
Figure 8: Additional attention visualizations across ViT layers. Supplementary heatmaps showing the progression of attention patterns from shallow to deep ViT layers.
Figure 9: Additional qualitative comparison examples. More case studies comparing DGP with LLaVA-1.5 across diverse visual understanding tasks.
Figure 10: Additional feature visualizations (Part 1). Supplementary DGP gated-fusion feature maps across all 32 LLM layers, illustrating improved spatial structure and semantic coherence.
Figure 11: Additional feature visualizations (Part 2). More comparisons of fixed single-layer features, uniform average fusion, and DGP ’s gated fusion at different LLM layers.
Figure 12: Additional question-aware routing analysis. More examples showing how routing weights shift as the question varies for the same image.
Figure 13: Routing weight distributions at LLM layer 32. Weights that the last LLM layer assigns across ViT layers on different benchmarks, showing that distinct tasks favor different ViT-layer combinations.
Multimodal large language models (MLLMs) commonly inherit the deep, symmetric Transformer backbone designed for unimodal text modeling, and apply the same computation uniformly to image and language tokens. This design overlooks a key modality asymmetry: image and text tokens differ substantially in information density, redundancy, and required reasoning depth. Through a layer-wise analysis of LLaVA-1.5, we observe that vision tokens tend to saturate in the middle layers. Specifically, text-to-image attention decreases from 0.68 at layer 0 to 0.07 by layer 4, and stabilizes near 0.04 after layer 18, whereas text tokens continue to benefit from deep semantic processing. These findings suggest a mismatch between architectural symmetry and depth-asynchronous modality evolution, resulting in redundant visual computation and possible drift in perceptual representations during deep task-specific adaptation. Motivated by this, we propose Dual-Path Vision Token Routing (DPVR), a modality-asymmetric routing framework for efficient MLLMs. Its core instantiation, DPVR-LF (Late-Layer Fusion), routes vision tokens at the saturation point into a one-layer trainable side branch, runs a thirteen-layer text-only forward that skips image positions in the deep stack, and re-fuses the visual and textual streams only at the final layer. With approximately 3% trainable parameters, DPVR-LF preserves competitive multimodal performance on standard benchmarks while reducing visual computation in the deep Transformer stack. The results challenge the conventional assumption that vision tokens must traverse all deep language-model layers, and indicate that a single late fusion layer can be sufficient for maintaining strong perceptual competence in LLaVA-style MLLMs.
Siyuan Liu, Jinyang Wu
aSchool of Mechanics and Engineering Science, Peking University, China · bDepartment of Automation, Tsinghua University, China
Pre-trained vision encoders contain layer-wise visual representations that differ in spatial granularity, semantic abstraction, and sensitivity to local details. However, most Multimodal Large Language Models (MLLMs) rely on only the final or penultimate vision encoder representations or fixed aggregation rules, making visual abstraction largely query-agnostic and limiting access to fine-grained cues such as small objects, spatial details, text, and subtle visual attributes. In this work, we propose Mixture of Layers (MoL), an instruction-conditioned layer routing approach at the visual patch level that dynamically aggregates query-relevant latent representations from intermediate vision encoder layers. Given a text query, MoL predicts routing probabilities over vision encoder layers and performs a top-k sparse aggregation over selected hidden states at either the image level, patch level, or through a hybrid routing mechanism. In doing so, MoL enables query-adaptive access to layer-specific visual features for fine-grained visual reasoning. Our experiments across 7 fine-grained visual reasoning tasks demonstrate substantial performance improvements, especially across fine-grained visual grounding and understanding tasks such as +18.9% improvement on V* in overall accuracy, +4.5% on HRBench4K, and +16.3% on CharXiv compared to the baseline MLLMs, without resorting to multi-resolution inputs, simple interleaving of multiple vision encoders, or increasing the number of patch tokens. We study vision encoders' receptive field scales across different layers and their sampling behaviors to provide an in-depth analysis of why layer-wise sampling is helpful, demonstrating that conditional visual representations are a key step towards better visual perception and reasoning in MLLMs. Our project page is available at https://wjdghks950.github.io/mol.github.io/.
Jeonghwan Kim, Sofia Stoica, Jiwan Chung +5
University of Illinois Urbana-Champaign · Yonsei University · Meta Reality Labs
Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection that links only the output of the vision encoder to the input of the large language model (LLM). This static architecture fundamentally limits the ability of LLMs to achieve comprehensive alignment with hierarchical visual knowledge, compromising their capacity to accurately integrate local details with global semantics into coherent reasoning. To resolve this, we introduce Cross-Layer Injection (CLI), a novel and lightweight framework that forges a dynamic many-to-many bridge between the two modalities. CLI consists of two synergistic, parameter-efficient components: an Adaptive Multi-Projection (AMP) module that harmonizes features from diverse vision layers, and an Adaptive Gating Fusion (AGF) mechanism that empowers the LLM to selectively inject the most relevant visual information based on its real-time decoding context. We validate the effectiveness and versatility of CLI by integrating it into LLaVA-OneVision and LLaVA-1.5. Extensive experiments on 18 diverse benchmarks demonstrate significant performance improvements, establishing CLI as a scalable paradigm that unlocks deeper multimodal understanding by granting LLMs on-demand access to the full visual hierarchy.
Cheng Chen, Yuyu Guo, Pengpeng Zeng +4
Tongji University, China · Ant Group, China · Shanghai Innovation Institute, China +1