cs.CVMar 9, 2026
SaveGranulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM
Organizations: National University of Singapore · Nanyang Technological University
Abstract
Recent advances in multimodal large language models largely rely on CLIP-based visual encoders, which emphasize global semantic alignment but struggle with fine-grained visual understanding. In contrast, DINOv3 provides strong pixel-level perception yet lacks coarse-grained semantic abstraction, leading to limited multi-granularity reasoning. To address this gap, we propose Granulon, a novel DINOv3-based MLLM with adaptive granularity augmentation. Granulon introduces a text-conditioned granularity Controller that dynamically adjusts the visual abstraction level according to the semantic scope of the textual input, and an Adaptive Token Aggregation module that performs granularity-guided pooling and relation-aware clustering to produce compact, semantically rich visual tokens. This design enables unified "pixel-to-fine-to-coarse" reasoning within a single forward pass. Extensive and interpretable experiments demonstrate that Granulon improves accuracy by ~30% and reduces hallucination by ~20%, outperforming all visual encoders under identical settings.
Figures & tables
Figure 1: CLIP tends to emphasize global semantics and DINOv3 excels in pixel-level understanding. Our Granulon unleashes the potential of DINOv3 with adaptive granularity augmentation, achieving consistent semantic alignment across different levels of detail and delivering superior multimodal reasoning performance.
Figure 2: Overview of Granulon . (a) The architecture of the image processor. (b) The detailed process of AdaTA that generates semantic tokens for multi-granularity augmentation. (c) The training and running of Controller.
| Encoders | Models | SEED-Bench | A-OKVQA | Image Caption | Reasoning | ||||
| Recall(%) | GPTscore(%) | Recall(%) | GPTscore(%) | BERTscore(%) | GPTscore(%) | BERTscore(%) | GPTscore(%) | ||
| CLIP | Qwen2.5 | 50.91_{\color[rgb]{0,0.88,0}{\uparrow 7.89}} | 24.4_{\color[rgb]{0,0.88,0}{\uparrow 11.4}} | 21.79_{\color[rgb]{0,0.88,0}{\uparrow 35.34}} | 10.3_{\color[rgb]{0,0.88,0}{\uparrow 25.6}} | \mathbf{84.35}_{\color[rgb]{1,0,0}{\downarrow 0.04}} | 21.54_{\color[rgb]{0,0.88,0}{\uparrow 9.74}} | 87.79_{\color[rgb]{0,0.88,0}{\uparrow 1.15}} | 29.36_{\color[rgb]{0,0.88,0}{\uparrow 19.95}} |
| SigLIP | Qwen2.5 | 46.72_{\color[rgb]{0,0.88,0}{\uparrow 12.08}} | 23.1_{\color[rgb]{0,0.88,0}{\uparrow 12.7}} | 21.89_{\color[rgb]{0,0.88,0}{\uparrow 35.24}} | 5.1_{\color[rgb]{0,0.88,0}{\uparrow 30.8}} | 83.92_{\color[rgb]{0,0.88,0}{\uparrow 0.39}} | 13.59_{\color[rgb]{0,0.88,0}{\uparrow 17.69}} | 86.51_{\color[rgb]{0,0.88,0}{\uparrow 2.43}} | 23.59_{\color[rgb]{0,0.88,0}{\uparrow 25.72}} |
| DINOv2 | Qwen2.5 | 41.40_{\color[rgb]{0,0.88,0}{\uparrow 17.40}} | 17.7_{\color[rgb]{0,0.88,0}{\uparrow 18.1}} | 16.67_{\color[rgb]{0,0.88,0}{\uparrow 40.46}} | 14.1_{\color[rgb]{0,0.88,0}{\uparrow 21.8}} | 83.40_{\color[rgb]{0,0.88,0}{\uparrow 0.91}} | 14.36_{\color[rgb]{0,0.88,0}{\uparrow 16.92}} | 88.15_{\color[rgb]{0,0.88,0}{\uparrow 0.79}} | 36.67_{\color[rgb]{0,0.88,0}{\uparrow 12.64}} |
| DINOv3 | Qwen2.5 | 55.74_{\color[rgb]{0,0.88,0}{\uparrow 3.06}} | 15.4_{\color[rgb]{0,0.88,0}{\uparrow 20.4}} | 47.43_{\color[rgb]{0,0.88,0}{\uparrow 9.70}} | 25.6_{\color[rgb]{0,0.88,0}{\uparrow 10.3}} | 83.90_{\color[rgb]{0,0.88,0}{\uparrow 0.41}} | 23.97_{\color[rgb]{0,0.88,0}{\uparrow 7.31}} | 88.26_{\color[rgb]{0,0.88,0}{\uparrow 0.68}} | 45.31_{\color[rgb]{0,0.88,0}{\uparrow 4.00}} |
| Ours | Qwen2.5 | ||||||||
Table 1: Comparison of various visual encoders and LLMs across different datasets. Best results are bold .
| Encoders | Phase Recognition | Instrument Recognition | ||
| BERTscore(%) | Recall(%) | BERTscore(%) | Recall(%) | |
| CLIP | 91.64_{\color[rgb]{0,0.88,0}{\uparrow 5.68}} | 46.15_{\color[rgb]{0,0.88,0}{\uparrow 30.77}} | 94.44_{\color[rgb]{0,0.88,0}{\uparrow 3.51}} | 46.92_{\color[rgb]{0,0.88,0}{\uparrow 29.15}} |
| SigLIP | 91.98_{\color[rgb]{0,0.88,0}{\uparrow 5.34}} | 48.72_{\color[rgb]{0,0.88,0}{\uparrow 28.20}} | 95.71_{\color[rgb]{0,0.88,0}{\uparrow 2.04}} | \mathbf{78.20}_{\color[rgb]{1,0,0}{\downarrow 2.13}} |
| DINOv2 | 91.67_{\color[rgb]{0,0.88,0}{\uparrow 5.65}} | 33.33_{\color[rgb]{0,0.88,0}{\uparrow 43.59}} | 95.93_{\color[rgb]{0,0.88,0}{\uparrow 2.02}} | 55.21_{\color[rgb]{0,0.88,0}{\uparrow 20.86}} |
| DINOv3 | 94.71_{\color[rgb]{0,0.88,0}{\uparrow 2.61}} | 64.10_{\color[rgb]{0,0.88,0}{\uparrow 12.82}} | 97.41_{\color[rgb]{0,0.88,0}{\uparrow 0.54}} | 68.89_{\color[rgb]{0,0.88,0}{\uparrow 7.18}} |
| Ours | ||||
Table 2: Comparison of various visual encoders across different medical tasks. Best results are bold .
Figure 3: Distribution of accuracy and granularity obtained from reasoning outputs. We summarize results across 120 samples, where the granularity and accuracy scores of each sentence are collected. Lower granularity score means finer-grained outputs.
Figure 4: Hallucination rate of various visual encoder.
Figure 5: Comparison of sentence-level granularity and hallucination scores across different visual encoders. Sentence index means the n-th sentence generated by LLMs.
Figure 6: Performance Comparison of multi-scale granularity.
Figure 7: Visualization of the performance metrics and input token consumption of different scale.
Figure 8: We first only input the reference texts into the LLM to obtain the hidden states at each layer, and then compute the cosine similarity between the reference state and the state of each corresponding layer for both CLIP baseline and Granulon .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Cosine similarity between the reference state and the state of each corresponding layer for CLIP +Llama3.
Figure 10: Cosine similarity between the reference state and the state of each corresponding layer for CLIP +Qwen2.5.
Figure 11: Cosine similarity between the reference state and the state of each corresponding layer for DINOV2 +Llama3.
Figure 12: Cosine similarity between the reference state and the state of each corresponding layer for DINOV2 +Qwen2.5.
Figure 13: Cosine similarity between the reference state and the state of each corresponding layer for DINOV3 +Llama3.
Figure 14: Cosine similarity between the reference state and the state of each corresponding layer for DINOV3 +Qwen2.5.
Figure 15: Cosine similarity between the reference state and the state of each corresponding layer for SigLIP +Llama3.
Figure 16: Cosine similarity between the reference state and the state of each corresponding layer for SigLIP +Qwen2.5.
Figure 17: Cosine similarity between the reference state and the state of each corresponding layer for Granulon +Llama3.
Figure 18: Cosine similarity between the reference state and the state of each corresponding layer for Granulon +Qwen2.5.