A central yet under-explored question in Multimodal Large Language Models (MLLMs) is where to connect---how visual information from a hierarchical vision encoder should be wired into the layer-wise semantics of a large language model. Mainstream MLLMs inject a fixed single-layer visual representation into the LLM and treat all LLM layers as a single consumer, ignoring that different layers demand different visual granularities. Recent multi-layer fusion methods either compress hierarchical features on the encoder side or rely on predefined sparse or hierarchical layer-to-layer connections, and therefore stop short of modeling adaptive cross-layer interactions between the ViT and LLM hierarchies. We address this gap with DGP (Dynamic Gated Pathways), a routing mechanism that establishes all-to-all connections between every ViT layer and every LLM layer, with each LLM layer dynamically gating its preference over visual layers conditioned on its own semantic state. Built on top of LLaVA-1.5-7B and 13B, DGP delivers consistent gains over the LLaVA-1.5-7B baseline (e.g., +4.2 on SQAI, +3.3 on MM-Vet, +3.1 on VizWiz and LLaVAW) and surpasses prior multi-layer visual fusion methods on most benchmarks. Layer-wise routing analyses further show that different LLM layers do select different visual granularities on demand, supporting both the effectiveness and interpretability of the proposed dynamic gated pathways. The code is available: https://github.com/YingyingYan1/DGP.
Figures & tables
Figure 1: Attention distributions across ViT layers. Shallow layers show dispersed attention over edges and textures; deeper layers progressively concentrate on a few semantic anchors.
Figure 2: Strategies for visual token insertion. (A) Fixed single-layer connection. (B) Encoder-side fusion of multiple ViT layers into one representation. (C) Predefined sparse or hierarchical connections between ViT and LLM layers. (D) DGP (Ours): dynamic gated pathways between all ViT and all LLM layers. wm denotes the routing weights that LLM layer m assigns across all ViT layers.
Figure 3: Overview of DGP . (A) At every LLM layer, the Dynamic Gated Pathways module routes over all ViT layers conditioned on the current language-layer state, aggregates the selected visual features, and injects them back into the LLM hidden states through a gated residual connection—establishing all-to-all cross-layer connections between ViT and LLM. (B) Inside the module, ViT layer summaries serve as keys/values and the language-layer state as the query; a shared cross-attention followed by a layer-specific routing MLP produces the per-layer routing weights. Weight Matrix (m) denotes the routing weights of LLM layer m over all ViT layers.
Academic-oriented VQA
LMM benchmarks
Method
LLM
GQA
SQA I
VQA v2
VizWiz
SEED
SEED I
LLaVA W
MM-Vet
MMB
MMB CN
POPE
BLIP-2 [ 19 ]
Vicuna-13B
41.0
61.0
65.0
19.6
46.4
49.7
38.1
22.4
–
–
85.3
InstructBLIP [ 11 ]
Vicuna-7B
49.2
60.5
–
34.5
53.4
58.8
60.9
26.2
36.0
23.7
–
InstructBLIP [ 11 ]
Vicuna-13B
49.5
63.1
–
33.4
78.9
–
58.2
25.6
–
–
78.9
Shikra [ 9 ]
Vicuna-13B
–
–
77.4
–
–
–
–
–
58.8
–
–
IDEFICS-9B [ 17 ]
LLaMA-7B
38.4
–
50.9
35.5
–
44.5
–
–
48.2
25.2
–
Table 1: Comparison with state-of-the-art MLLMs. We evaluate on Academic-oriented VQA tasks and recent LMM benchmarks. Best results are in bold ; second-best are underlined . “ -- ” denotes results not reported in the original papers. Rows below DGP report the absolute point gain ( ↑ ) or loss ( ↓ ) relative to LLaVA-1.5.
Figure 4: Qualitative comparison between DGP and LLaVA-1.5. Representative examples from math solving, OCR understanding, and detailed visual reasoning. LLaVA-1.5 often misses critical visual evidence or hallucinates content; DGP produces more accurate, evidence-grounded responses by adaptively accessing multi-granularity visual features.
Avg.
Gated
Entropy
Academic-oriented VQA
LMM benchmarks
Fusion
Routing
Reg.
GQA
SQA I
VQA v2
VizWiz
SEED
SEED I
LLaVA W
MM-Vet
MMB
MMB CN
POPE
✗
✗
✗
62.0
66.8
78.5
50.0
58.6
66.1
63.4
30.5
64.3
58.3
85.9
✓
✗
✗
60.2
64.3
72.0
50.2
57.9
63.3
56.3
25.3
57.6
44.9
77.2
✗
✓
✗
64.1
70.5
78.9
52.0
60.9
66.7
63.6
32.4
65.6
59.1
86.4
✗
✓
✓
64.6
71.0
79.7
53.1
61.5
67.5
66.5
33.8
66.7
60.5
87.0
Table 4: Ablation of DGP components. We evaluate the contribution of average fusion, gated routing, and entropy regularization.
Figure 5: Gated Pathway Analysis. (A) Heatmap of routing weights over LLM (x) and ViT (y) layers. (B) Routing curves of representative LLM layers with multimodal patterns. (C) Routing weights of LLM layer 21 across benchmarks, showing task-specific preferences. (D) Per-sample routing distributions within the same benchmark, showing sample-level diversity.
Figure 6: Task-aware routing for the same image. As the question changes, routing shifts: attribute recognition activates shallower layers, while global-semantic questions activate deeper ones.
Figure 7: Fused visual feature visualization. We compare CLIP layer 23, the uniform average over all 24 ViT layers, and DGP ’s gated fusion at different LLM layers.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Operation
Tensor Shape
Description
Vision Encoding (ViT)
Raw patches
ViT layer ℓ output
[B,577,1024]
1 CLS + 576 patches per layer
Patch features
Remove CLS
[B,576,1024]×24
Cache all 24 layers for routing
CLS tokens
Extract CLS
[B,1,1024]×24
Layer summaries for cross-attention
CLS Projection for Routing
CLS projection
LN(1024)→Linear(1024→4096)
[B,1,4096]×24
Project each CLS to LLM space
Appendix
Table 5: Token flow and tensor dimensions in DGP . We illustrate each processing stage using Vicuna-7B (32 layers, d=4096 ) and CLIP-ViT-L/14-336 (24 layers, dv=1024 , Nv=576 patches). B denotes batch size, Nt denotes the number of text tokens.
Pre-training (PT)
Supervised Fine-tuning (SFT)
Global batch size
256
Global batch size
128
Learning rate
1×10−3
Learning rate
2×10−5
Vision encoder LR
frozen
Vision encoder LR
frozen
LR schedule
cosine decay
LR schedule
cosine decay
LR warmup ratio
0.03
LR warmup ratio
0.03
Epochs
1
Epochs
1
Appendix
Table 6: Training hyperparameters. Configuration for pre-training (PT) on LLaVA-558K and supervised fine-tuning (SFT) on LLaVA-mixed-665K.
Category
Benchmark
Task Type
Evaluated Capabilities
Academic VQA
VQA v2 [ 13 ]
General VQA
Basic image understanding, object attribute recognition, and commonsense visual reasoning
GQA [ 15 ]
Compositional VQA
Compositional visual reasoning with structured scene-graph annotations (relations, attributes, and logical consistency)
VizWiz [ 14 ]
Real-world VQA
Real user questions from visually impaired users with often noisy images, testing robust visual understanding
SQA I [ 29 ]
Science QA (image)
Scientific knowledge, image-grounded reasoning, and multi-step inference
LMM Bench
POPE [ 22 ]
Hallucination probing
Contrastive image–question pairs to test visual grounding and robustness against object hallucination
MMBench (MMB/MMB CN ) [ 28 ]
Multi-choice
Classification-style questions covering perception, reasoning, and instruction following (English/Chinese subsets)
Appendix
Table 7: Benchmark descriptions. We evaluate on two complementary suites: academic task-oriented benchmarks and instruction-following / LMM benchmarks.
γinit
Academic-oriented VQA
LMM benchmarks
GQA
SQA I
VQA v2
VizWiz
SEED
SEED I
LLaVA W
MM-Vet
MMB
MMB CN
POPE
0.1
64.4
70.6
79.3
52.7
61.4
67.3
66.2
33.5
66.2
60.2
87.2
0.3 (default)
64.6
71.0
79.7
53.1
61.5
67.5
66.5
33.8
66.7
60.5
87.0
0.5
64.3
70.8
79.1
53.0
61.4
67.1
66.3
34.0
66.5
60.4
86.9
Appendix
Table 8: Effect of gating initialization γinit . We vary the initial value of the residual gate and report performance across the full benchmark suite.
Component
7B (M)
% of DGP
13B (M)
% of DGP
Routers (per-layer MLP)
135.3
37.80%
211.2
37.88%
Shared Cross-Attention
67.1
18.75%
104.9
18.81%
CLS Expert
4.2
1.17%
5.3
0.95%
Post-Fusion Experts
134.4
37.55%
210.0
37.66%
Others (embeddings, gates, norms)
16.9
4.72%
26.2
4.70%
Total DGP Module
357.9
100%
557.6
100%
Appendix
Table 9: Parameter breakdown of the DGP module.
Academic-oriented VQA
LMM benchmarks
Method
LLM
GQA
SQA I
VQA v2
VizWiz
SEED
SEED I
LLaVA W
MM-Vet
MMB
MMB CN
POPE
LLaVA-1.5 [ 25 ]
Vicuna-7B
62.0
66.8
78.5
50.0
58.6
66.1
63.4
30.5
64.3
58.3
85.9
DGP
Vicuna-7B
64.6
71.0
79.7
53.1
61.5
67.5
66.5
33.8
66.7
60.5
87.0
DGP ‡
Vicuna-7B
64.1
71.0
78.9
50.2
59.7
65.3
63.5
33.7
68.3
59.4
84.8
Appendix
Table 10: Effect of unfreezing the vision encoder. Comparison between DGP with frozen and unfrozen ( ‡ ) vision encoder during training.
LLM
ViT Layer
Layer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
1
2.4
2.1
1.9
2.9
16.8
2.2
38.3
2.7
5.8
6.8
2.8
2.2
1.8
1.5
1.6
1.4
1.3
1.1
0.9
0.8
0.7
0.6
0.6
0.6
2
2.0
2.0
1.8
2.6
2.3
2.3
2.7
4.6
4.8
5.1
6.3
4.0
19.1
12.1
10.2
9.5
2.8
1.6
1.0
0.8
0.7
0.6
0.5
0.6
3
1.7
1.5
1.3
1.3
1.4
1.3
1.5
1.5
1.8
1.9
2.1
11.0
58.8
2.5
2.5
1.7
1.4
1.1
1.0
0.7
0.6
0.5
0.4
0.5
4
1.8
1.8
1.7
1.9
1.9
2.0
3.0
3.5
4.3
2.9
3.4
11.6
44.9
2.6
3.3
2.7
1.7
1.2
1.0
0.8
0.6
0.5
0.4
0.5
5
1.7
1.7
1.6
1.6
1.5
1.6
1.8
2.0
3.2
2.6
4.1
20.8
11.3
29.9
3.8
3.2
1.9
1.5
1.0
0.8
0.7
0.6
0.5
0.7
Appendix
Table 11: Full routing weight matrix on POPE. Each cell shows the percentage weight ( ×100 ) that LLM layer l assigns to ViT layer v . Rows sum to 100.
Routing
Academic-oriented VQA
LMM benchmarks
GQA
SQA I
VQA v2
VizWiz
SEED
SEED I
LLaVA W
MM-Vet
MMB
MMB CN
POPE
DGP (Learned)
64.6
71.0
79.7
53.1
61.5
67.5
66.5
33.8
66.7
60.5
87.0
Baseline (Last-2)
62.0
66.8
78.5
50.0
58.6
66.1
63.4
30.5
64.3
58.3
85.9
Uniform
60.2
64.3
72.0
50.2
57.9
63.3
56.3
25.3
57.6
44.9
77.2
Reversed
63.6
67.5
77.8
52.4
61.3
67.1
57.0
32.1
61.7
55.8
86.6
Appendix
Table 12: Inference-time routing intervention. Comparison of different routing strategies on DGP-7B without retraining.
Figure 8: Additional attention visualizations across ViT layers. Supplementary heatmaps showing the progression of attention patterns from shallow to deep ViT layers.
Figure 9: Additional qualitative comparison examples. More case studies comparing DGP with LLaVA-1.5 across diverse visual understanding tasks.
Figure 10: Additional feature visualizations (Part 1). Supplementary DGP gated-fusion feature maps across all 32 LLM layers, illustrating improved spatial structure and semantic coherence.
Figure 11: Additional feature visualizations (Part 2). More comparisons of fixed single-layer features, uniform average fusion, and DGP ’s gated fusion at different LLM layers.
Figure 12: Additional question-aware routing analysis. More examples showing how routing weights shift as the question varies for the same image.
Figure 13: Routing weight distributions at LLM layer 32. Weights that the last LLM layer assigns across ViT layers on different benchmarks, showing that distinct tasks favor different ViT-layer combinations.