The field of computer vision has experienced significant advancements through scalable vision encoders and multimodal pre-training frameworks. However, existing approaches often treat vision encoders and large language models (LLMs) as independent modules, limiting the integration of hierarchical visual features. In this work, we propose HIVE (Hierarchical Pre-Training of Vision Encoders), a novel framework that enhances vision-language alignment by introducing hierarchical cross-attention between the vision encoder and LLM. Unlike conventional methods that flatten image embeddings, HIVE enables structured feature fusion across multiple layers, improving gradient flow and representation learning. To optimize this interaction, we introduce a three-stage training strategy that progressively aligns the vision encoder with the LLM, ensuring stable optimization and effective multimodal fusion. Empirical evaluations demonstrate that HIVE achieves superior performance not only in image classification but also on various vision-language tasks, outperforming self-attention-based methods in benchmarks such as MME, GQA, OK-VQA, and ScienceQA. Our results highlight the benefits of hierarchical feature integration, paving the way for more efficient and expressive vision-language models.
Figures & tables
Figure 1: Overview of the proposed Hierarchical Pre-Training of Vision Encoders (HIVE) framework. The vision encoder extracts hierarchical visual features, which are projected and integrated into the large language model (LLM) through cross-attention. This enables multi-layered feature alignment, improving representation learning, semantic understanding, and gradient flow. Green text indicates correct model predictions, while red text highlights incorrect predictions.
Method
CIFAR-10
CIFAR-100
ImageNet-1K
Tiny-ImageNet
Food-101
Cars
Pets
Caltech-256
CLIP-based Models
Base
98.32 ± 0.04
87.92 ± 0.08
84.01 ± 0.01
86.65 ± 0.11
95.75 ± 0.03
91.17 ± 0.04
96.02 ± 0.10
96.12 ± 0.08
SA
98.37 ± 0.01
87.93 ± 0.06
84.14 ± 0.02
86.51 ± 0.04
95.77 ± 0.03
91.04 ± 0.13
95.98 ± 0.05
96.27 ± 0.10
HIVE
98.49 ± 0.01
88.49 ± 0.07
84.08 ± 0.00
86.71 ± 0.08
95.78 ± 0.03
91.15 ± 0.23
95.92 ± 0.07
96.35 ± 0.11
SigLIP-based Models
Base
98.43 ± 0.01
90.08 ± 0.11
85.99 ± 0.02
82.27 ± 0.20
96.55 ± 0.01
94.86 ± 0.15
96.76 ± 0.04
97.27 ± 0.09
Table 1: Classification accuracy (%) on various benchmarks comparing our proposed method, HIVE, with self-attention-based approaches. Results are reported as mean ± standard deviation (SD) over 3 runs. Base represents the original foundation models (CLIP or SigLIP) without additional LLM-supported pre-training. SA represents a self-attention-based vision encoder with LLM-supported pre-training, while HIVE represents our proposed cross-attention-based pre-training framework. The best results are in bold.
Encoder
MME
GQA
OK-VQA
ScienceQA
Base
1296
57.74
48.78
62.34
SA
1263
57.69
46.19
59.56
HIVE
1298
58.05
51.01
63.12
Table 2: LLM-based evaluation results on vision-language tasks using SigLIP. Both SA and HIVE follow our three-stage pre-training method for the vision encoder. The final VLM training stage follows the procedure used in LLaVA. The best results are in bold.
Figure 2: Gradient map visualization from the first to the final layer. The first image is the original image with the caption: “a horse rider in full riding gear is riding on a horse - stock photo.” The remaining images demonstrate more granular gradients in earlier layers due to cross-attention pre-training.
Figure 3: Attention map visualization illustrating hierarchical cross-attention behavior. The x-axis corresponds to sampled tokens, while the y-axis represents encoder layers from lower (top) to higher (bottom) layers. HIVE efficiently integrates multi-level features, with lower layers attending to specific details and higher layers producing broader, more abstract activations.
Figure 4: Comparison of hierarchical and cascaded pre-training approaches for vision encoders. Hierarchical pre-training (left) establishes direct cross-attention across multiple layers, allowing for stronger gradient propagation and better feature integration. Cascaded pre-training (right) restricts interactions to the final layer, leading to attenuated gradient signals and weaker hierarchical feature learning.
Stage
Stage 1 (Projector)
Stage 2 (Projector + LLM)
Stage 3 (Full Model)
Optimizer
Fully decoupled AdamW [ 25 ]
Optimizer Momentum
β1=0.9,β2=0.999
β1=0.9,β2=0.95
β1=0.9,β2=0.95
Peak learning rate
1×10−3
2×10−5
2×10−6
Minimum learning rate
1×10−4
2×10−6
0
Weight decay
0
0
0
Batch size
256
256
1024
Table 3: Pre-training hyperparameters for the three-stage pre-training procedure used in HIVE.
Config
Setting
Optimizer
AdamW [ 25 ]
Optimizer Momentum
β1=0.9,β2=0.999
Peak learning rate grid
2×10−4
Minimum learning rate
0
Weight decay
0.01
Batch size
512
Table 4: Classifier fine-tuning hyperparameters for HIVE and baselines.
Config
Stage 1 (Connector Training)
Stage 2 (LLM Fine-Tuning)
Optimizer
AdamW [ 25 ]
AdamW [ 25 ]
Optimizer Momentum
β1=0.9,β2=0.999
β1=0.9,β2=0.999
Peak learning rate
1×10−3
2×10−5
Minimum learning rate
0
0
Weight decay
0
0
Batch size
256
64
Table 5: Hyperparameters for VLM fine-tuning on vision-language tasks.
Method
Relative Efficiency
Training Cost (TFLOPs)
Self-Attention
1.0×
HIVE (Ours)
0.14×
Memory Overhead
Self-Attention
1.0×
HIVE (Ours)
0.8×
Table 6: Computational efficiency comparison between self-attention and HIVE. Values are reported relative to the self-attention baseline.
Method
Complexity
Self-Attention (Causal)
O(Ll2N2d+LlNd2)
Cross-Attention (Causal)
O(LlLsd2+LlNtd2)
Table 7: Computational complexity comparison between full self-attention and hierarchical cross-attention in vision-language models.
Model
Training Time (min/epoch)
Speedup
Self-Attention
240
-
Cross-Attention
70
3.43 ×
Table 8: Average wall-clock training time per epoch. Hierarchical cross-attention provides notable speedups over self-attention.
Figure 5: Gradient map visualization for Sample 1: ”Investigators and journalists gather around the car of person after an attack on Wednesday.” Gradients emphasize individuals and vehicles with sharp, localized activations in early layers.
Figure 6: Gradient map visualization for Sample 2: ”Colorful plastic and aluminum chairs leaning against tables at a cafe outdoor dining area.” Early layers highlight fine-grained details such as chair edges, while deeper layers emphasize broader scene structure.
Figure 7: Gradient map visualization for Sample 3: ”Person makes a move on defenders during the spring game.” HIVE captures dynamic motion cues, focusing on the athlete and defenders.
Figure 8: Gradient map visualization for Sample 4. HIVE maintains granular focus on key visual features, improving gradient flow.
Figure 9: Gradient map visualization for Sample 5. Enhanced gradient stability enables sharper feature refinement in early layers.
Figure 10: Gradient map visualization for Sample 6. HIVE consistently emphasizes meaningful visual elements across encoder layers.
Figure 11: Gradient map visualization for Sample 7. Stable gradient propagation ensures effective visual feature learning across hierarchical layers.
Figure 12: Attention map visualization for Sample 1: ”Investigators and journalists gather around the car of person after an attack on Wednesday.” HIVE emphasizes key elements such as the car and surrounding individuals, demonstrating improved semantic localization.
Figure 13: Attention map visualization for Sample 2: ”Colorful plastic and aluminum chairs leaning against tables at a cafe outdoor dining area.” HIVE highlights chairs and tables with sharper focus, enhancing object-level localization.
Figure 14: Attention map visualization for Sample 3: ”Person makes a move on defenders during the spring game.” HIVE effectively highlights the athlete’s movement and surrounding players, improving focus on dynamic elements.
Multimodal Large Language Models (MLLMs) have achieved remarkable performance by aligning pretrained visual representations with the linguistic knowledge embedded in Large Language Models (LLMs). However, existing approaches typically rely on final-layer visual features or learnable multi-layer fusion, which often fail to sufficiently exploit hierarchical visual cues without explicit cross-layer interaction design. In this work, we propose a Memory-Augmented Adapter (Mema) within the vision encoder. Specifically, Mema maintains a stateful memory that accumulates hierarchical visual representations across layers, with its evolution conditioned on both query embeddings and step-wise visual features. A portion of this memory is selectively injected into token representations via a feedback mechanism, thereby mitigating the attenuation of fine-grained visual cues from shallow layers. Designed as a lightweight and plug-and-play module, Mema integrates seamlessly into pretrained vision encoders without modifying the vanilla backbone architecture. Only a minimal set of additional parameters requires training, enabling adaptive visual feature refinement while reducing training overhead. Extensive experiments across multiple benchmarks demonstrate that Mema consistently improves performance, validating its effectiveness in complex multimodal reasoning tasks. The code have been released at https://github.com/Sisiliu312/Mema.
Ying Liu, Yudong Han, Kean Shi +1
Beijing Institute of Technology · Peking University · Yangtze Delta Region Academy of Beijing Institude of Technology, Jiaxing, China
Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection that links only the output of the vision encoder to the input of the large language model (LLM). This static architecture fundamentally limits the ability of LLMs to achieve comprehensive alignment with hierarchical visual knowledge, compromising their capacity to accurately integrate local details with global semantics into coherent reasoning. To resolve this, we introduce Cross-Layer Injection (CLI), a novel and lightweight framework that forges a dynamic many-to-many bridge between the two modalities. CLI consists of two synergistic, parameter-efficient components: an Adaptive Multi-Projection (AMP) module that harmonizes features from diverse vision layers, and an Adaptive Gating Fusion (AGF) mechanism that empowers the LLM to selectively inject the most relevant visual information based on its real-time decoding context. We validate the effectiveness and versatility of CLI by integrating it into LLaVA-OneVision and LLaVA-1.5. Extensive experiments on 18 diverse benchmarks demonstrate significant performance improvements, establishing CLI as a scalable paradigm that unlocks deeper multimodal understanding by granting LLMs on-demand access to the full visual hierarchy.
Cheng Chen, Yuyu Guo, Pengpeng Zeng +4
Tongji University, China · Ant Group, China · Shanghai Innovation Institute, China +1
Recent vision-language models (VLMs) typically rely on a single vision encoder trained with contrastive image-text objectives, such as CLIP-style pretraining. While contrastive encoders are effective for cross-modal alignment and retrieval, self-supervised visual encoders often capture richer dense semantics and exhibit stronger robustness on recognition and understanding tasks. In this work, we investigate how to scale the fusion of these complementary visual representations for vision-language modeling. We propose CoME-VL: Complementary Multi-Encoder Vision-Language, a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder. Our approach performs representation-level fusion by (i) entropy-guided multi-layer aggregation with orthogonality-constrained projections to reduce redundancy, and (ii) RoPE-enhanced cross-attention to align heterogeneous token grids and produce compact fused visual tokens. The fused tokens can be injected into a decoder-only LLM with minimal changes to standard VLM pipelines. Extensive experiments across diverse vision-language benchmarks demonstrate that CoME-VL consistently outperforms single-encoder baselines. In particular, we observe an average improvement of 4.9% on visual understanding tasks and 5.4% on grounding tasks. Our method achieves state-of-the-art performance on RefCOCO for detection while improving over the baseline by a large margin. Finally, we conduct ablation studies on layer merging, non-redundant feature mixing, and fusion capacity to evaluate how complementary contrastive and self-supervised signals affect VLM performance.
Ankan Deria, Komal Kumar, Xilin He +4
Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE