From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception
Authors: Jilong Zhu, Yang Feng
Organizations: Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · University of Chinese Academy of Sciences, Beijing, China
While Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in general visual understanding, they frequently falter in fine-grained perception tasks that require identifying tiny objects or discerning subtle visual relationships. We attribute this limitation to Visual Attenuation: a phenomenon where sparse fine-grained visual signals are prematurely suppressed or diluted by dominant textual tokens during network propagation, resulting in a "loss of focus" during the deep-level decision-making process. Existing input-centric solutions fail to fundamentally reverse this intrinsic mechanism of information loss. To address this challenge, we propose the Variational Information Flow (VIF) framework. Adopting a probabilistic perspective, VIF leverages a Conditional Variational Autoencoder (CVAE) to model the visual saliency relevant to the question-answer pair as a latent distribution. As a plug-and-play module, VIF can be integrated into existing architectures. Extensive evaluations across diverse benchmarks, covering General VQA, fine-grained perception, and visual grounding, demonstrate that VIF yields competitive improvements over previous methods, validating its effectiveness in enhancing the fine-grained perception of MLLMs.
Figures & tables
Figure 1: Illustration of Visual Attenuation. (Left) Schematic view showing visual tokens (blue) fading as they propagate through deep layers, while textual tokens (green) dominate. (Right) Quantitative layer-to-layer changes in vision attention ratio, averaged over 500 randomly sampled instances. The sharp drop in early layers indicates a premature loss of visual details.
Figure 2: Layer-wise visual attention distribution. The red curve denotes the mean vision attention ratio across layers, which decreases sharply with depth. The violin plots illustrate the distribution of visual attention weights at each layer, with the long-tailed distribution shrinking in the deep layers.
Figure 3: Visualization of layer-wise visual attention. (Left) Shallow layers (L0-L1) capture dense contextual information with broad coverage. (Middle) Middle layers (L15-L16) successfully converge on key semantic regions (the board), exhibiting sparse and focused attention. (Right) In deep layers (L30-L31), this focus deteriorates into a diffuse and disordered state.
Figure 4: Overview of the Variational Information Flow (VIF) Framework. The framework consists of three stages: (1) Visual Signal Attenuation Analysis. The visual signal is out of focus in the deep layer. The model recovers rich visual cues from intermediate layers. (2) CVAE Probabilistic Attender Module. This module utilizes a GMM-based prior and posterior learning to reconstruct a sparse, task-relevant Spatial Attention Map from middle layers. (3) Visual Information Flow Restoration. This stage recovery involves injecting the learned visual focus into deeper layers to restore fine-grained visual cues to enhance the final prediction.
Figure 5: Examples of Vision-Language Task Samples. As illustrated, relying solely on questions often results in semantic ambiguity. Integrating answer information is thus critical to serve as a semantic anchor for robust task-driven visual modeling.
Model
LLM
Res.
Multimodal Understanding
Knowledge
OCR
Multi-Image
SEED-I
LLaVAB
MME
GQA
VQAv2
AI2D
SQA
TextVQA
BLINK
LLaVA-v1.5
Vicuna-7B
336×336
67.2
61.8
1480.6
61.9
78.5
55.5
67.1
58.1
39.7
InstructBLIP
Vicuna-7B
224×224
58.8
59.8
1137.1
49.2
-
40.6
60.5
50.1
-
mPLUG-Owl2
LLaMA 2-7B
448×448
64.5
59.9
1450.2
56.1
-
55.7
68.7
54.3
-
MiniGPT-v2
LLaMA 2-7B
448×448
31.6
45.1
770.6
60.1
-
28.4
39.6
-
-
LLaMA-Adapter-v2
LLaMA 2-7B
-
32.7
-
972.7
-
-
-
-
-
-
Table 1: Comparison on General Multimodal Capabilities. Bold denotes the best result, and underline denotes the second best.
Model
HR-Bench
Vstar
REC (RefCOCO)
4K
8K
val
testA
testB
LLaVA-v1.5
36.1
32.1
45.0
30.4
16.0
42.0
mPLUG-Owl2
36.9
33.8
35.6
-
-
-
MiniGPT-v2
25.5
26.1
-
-
-
-
IDEFICS-9B
30.6
28.8
-
-
-
-
Qwen-VL
31.8
28.4
-
-
-
-
Table 2: Fine-Grained Perception and Referring Expression Comprehension. Bold denotes the best result, and underline denotes the second best. REC results are reported with the CIDEr score.
Model
SEED-I
AI2D
HR-Bench
Vstar
REC (RefCOCO)
4K
8K
val
testA
testB
VIF (Ours)
70.8
64.0
44.8
36.8
50.8
34.3
16.6
47.0
- w/o AP
69.2
63.0
42.3
34.3
48.7
32.8
15.3
44.1
- w/o SP
68.2
62.5
36.8
32.9
49.2
29.6
13.9
40.7
Full-Seq
68.5
62.8
38.9
33.6
49.2
29.9
14.1
42.7
Deep-Only
68.2
63.1
39.1
33.0
49.7
30.7
15.6
43.1
Table 3: Ablation Study of the VIF Framework.
Figure 6: visualization of attention maps comparing the baseline and our proposed model. The top and bottom rows display the attention distributions of the baseline and our model, across different layers (L27 and L31).
Method / Configuration
Extraction Layers
Injection Layers
SEED-I
GQA
TextVQA
HR-Bench (4K)
Vstar
LLaVA-v1.5 (Baseline)
–
–
67.2
61.9
58.1
36.1
45.0
VIF (L-early, 4 pairs)
9,11,13,15
25,27,29,31
69.5
62.3
59.1
42.5
48.9
VIF (L-late, 4 pairs)
13,15,17,19
25,27,29,31
70.3
62.6
59.5
43.8
49.6
VIF (5 pairs)
9,11,13,15,17
23,25,27,29,31
70.9
62.9
59.8
45.0
50.9
VIF (Our Default)
11,13,15,17
25,27,29,31
70.8
62.8
59.9
44.8
50.8
Table 4: Ablation study on extraction to injection layer configurations.
AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) · International Digital Economy Academy · MedVisAI Lab, Lee Kong Chian School of Medicine, Nanyang Technological University, and Centre of AI in Medicine +2