Hierarchical Pre-Training of Vision Encoders with Large Language Model

Eugene Lee; Ting-Yu Chang; Jui-Huang Tsai; Jiajie Diao; Chen-Yi Lee

cs.CVMar 31, 2026

Hierarchical Pre-Training of Vision Encoders with Large Language Model

Authors: Eugene Lee, Ting-Yu Chang, Jui-Huang Tsai, Jiajie Diao, Chen-Yi Lee

Organizations: University of Cincinnati · National Yang Ming Chiao Tung University

Abstract

The field of computer vision has experienced significant advancements through scalable vision encoders and multimodal pre-training frameworks. However, existing approaches often treat vision encoders and large language models (LLMs) as independent modules, limiting the integration of hierarchical visual features. In this work, we propose HIVE (Hierarchical Pre-Training of Vision Encoders), a novel framework that enhances vision-language alignment by introducing hierarchical cross-attention between the vision encoder and LLM. Unlike conventional methods that flatten image embeddings, HIVE enables structured feature fusion across multiple layers, improving gradient flow and representation learning. To optimize this interaction, we introduce a three-stage training strategy that progressively aligns the vision encoder with the LLM, ensuring stable optimization and effective multimodal fusion. Empirical evaluations demonstrate that HIVE achieves superior performance not only in image classification but also on various vision-language tasks, outperforming self-attention-based methods in benchmarks such as MME, GQA, OK-VQA, and ScienceQA. Our results highlight the benefits of hierarchical feature integration, paving the way for more efficient and expressive vision-language models.

Figures & tables

Figure 1: Overview of the proposed Hierarchical Pre-Training of Vision Encoders (HIVE) framework. The vision encoder extracts hierarchical visual features, which are projected and integrated into the large language model (LLM) through cross-attention. This enables multi-layered feature alignment, improving representation learning, semantic understanding, and gradient flow. Green text indicates correct model predictions, while red text highlights incorrect predictions.

CLIP-based Models
Method	CIFAR-10	CIFAR-100	ImageNet-1K	Tiny-ImageNet	Food-101	Cars	Pets	Caltech-256
Base	98.32 $\pm$ 0.04	87.92 $\pm$ 0.08	84.01 $\pm$ 0.01	86.65 $\pm$ 0.11	95.75 $\pm$ 0.03	91.17 $\pm$ 0.04	96.02 $\pm$ 0.10	96.12 $\pm$ 0.08
SA	98.37 $\pm$ 0.01	87.93 $\pm$ 0.06	84.14 $\pm$ 0.02	86.51 $\pm$ 0.04	95.77 $\pm$ 0.03	91.04 $\pm$ 0.13	95.98 $\pm$ 0.05	96.27 $\pm$ 0.10
HIVE	98.49 $\pm$ 0.01	88.49 $\pm$ 0.07	84.08 $\pm$ 0.00	86.71 $\pm$ 0.08	95.78 $\pm$ 0.03	91.15 $\pm$ 0.23	95.92 $\pm$ 0.07	96.35 $\pm$ 0.11
SigLIP-based Models
Base	98.43 $\pm$ 0.01	90.08 $\pm$ 0.11	85.99 $\pm$ 0.02	82.27 $\pm$ 0.20	96.55 $\pm$ 0.01	94.86 $\pm$ 0.15	96.76 $\pm$ 0.04	97.27 $\pm$ 0.09

Table 1: Classification accuracy (%) on various benchmarks comparing our proposed method, HIVE, with self-attention-based approaches. Results are reported as mean ± standard deviation (SD) over 3 runs. Base represents the original foundation models (CLIP or SigLIP) without additional LLM-supported pre-training. SA represents a self-attention-based vision encoder with LLM-supported pre-training, while HIVE represents our proposed cross-attention-based pre-training framework. The best results are in bold.

Encoder	MME	GQA	OK-VQA	ScienceQA
Base	1296	57.74	48.78	62.34
SA	1263	57.69	46.19	59.56
HIVE	1298	58.05	51.01	63.12

Table 2: LLM-based evaluation results on vision-language tasks using SigLIP. Both SA and HIVE follow our three-stage pre-training method for the vision encoder. The final VLM training stage follows the procedure used in LLaVA. The best results are in bold.

Figure 2: Gradient map visualization from the first to the final layer. The first image is the original image with the caption: “a horse rider in full riding gear is riding on a horse - stock photo.” The remaining images demonstrate more granular gradients in earlier layers due to cross-attention pre-training.

Figure 3: Attention map visualization illustrating hierarchical cross-attention behavior. The x-axis corresponds to sampled tokens, while the y-axis represents encoder layers from lower (top) to higher (bottom) layers. HIVE efficiently integrates multi-level features, with lower layers attending to specific details and higher layers producing broader, more abstract activations.

Figure 4: Comparison of hierarchical and cascaded pre-training approaches for vision encoders. Hierarchical pre-training (left) establishes direct cross-attention across multiple layers, allowing for stronger gradient propagation and better feature integration. Cascaded pre-training (right) restricts interactions to the final layer, leading to attenuated gradient signals and weaker hierarchical feature learning.

Stage	Stage 1 (Projector)	Stage 2 (Projector + LLM)	Stage 3 (Full Model)
Optimizer	Fully decoupled AdamW [ 25 ]
Optimizer Momentum	$\beta_{1}=0.9,\beta_{2}=0.999$	$\beta_{1}=0.9,\beta_{2}=0.95$	$\beta_{1}=0.9,\beta_{2}=0.95$
Peak learning rate	$1\times 10^{-3}$	$2\times 10^{-5}$	$2\times 10^{-6}$
Minimum learning rate	$1\times 10^{-4}$	$2\times 10^{-6}$	$0$
Weight decay	$0$	$0$	$0$
Batch size	256	256	1024

Table 3: Pre-training hyperparameters for the three-stage pre-training procedure used in HIVE.

Config	Setting
Optimizer	AdamW [ 25 ]
Optimizer Momentum	$\beta_{1}=0.9,\beta_{2}=0.999$
Peak learning rate grid	$2\times 10^{-4}$
Minimum learning rate	$0$
Weight decay	$0.01$
Batch size	512

Table 4: Classifier fine-tuning hyperparameters for HIVE and baselines.

Config	Stage 1 (Connector Training)	Stage 2 (LLM Fine-Tuning)
Optimizer	AdamW [ 25 ]	AdamW [ 25 ]
Optimizer Momentum	$\beta_{1}=0.9,\beta_{2}=0.999$	$\beta_{1}=0.9,\beta_{2}=0.999$
Peak learning rate	$1\times 10^{-3}$	$2\times 10^{-5}$
Minimum learning rate	$0$	$0$
Weight decay	$0$	$0$
Batch size	256	64

Table 5: Hyperparameters for VLM fine-tuning on vision-language tasks.

Method	Relative Efficiency
Training Cost (TFLOPs)
Self-Attention	1.0×
HIVE (Ours)	0.14×
Memory Overhead
Self-Attention	1.0×
HIVE (Ours)	0.8×

Table 6: Computational efficiency comparison between self-attention and HIVE. Values are reported relative to the self-attention baseline.

Method	Complexity
Self-Attention (Causal)	$\mathcal{O}\left(L_{l}\frac{N^{2}d}{2}+L_{l}Nd^{2}\right)$
Cross-Attention (Causal)	$\mathcal{O}(L_{l}L_{s}d^{2}+L_{l}N_{t}d^{2})$

Table 7: Computational complexity comparison between full self-attention and hierarchical cross-attention in vision-language models.

Model	Training Time (min/epoch)	Speedup
Self-Attention	240	-
Cross-Attention	70	3.43 $\times$

Table 8: Average wall-clock training time per epoch. Hierarchical cross-attention provides notable speedups over self-attention.

Model	Peak Memory (GB)	Reduction
Self-Attention	54.2	-
Cross-Attention	22.03	59.3%

Table 9: Peak GPU memory usage during training. Hierarchical cross-attention reduces memory overhead substantially.

Figure 5: Gradient map visualization for Sample 1: ”Investigators and journalists gather around the car of person after an attack on Wednesday.” Gradients emphasize individuals and vehicles with sharp, localized activations in early layers.

Figure 6: Gradient map visualization for Sample 2: ”Colorful plastic and aluminum chairs leaning against tables at a cafe outdoor dining area.” Early layers highlight fine-grained details such as chair edges, while deeper layers emphasize broader scene structure.

Figure 7: Gradient map visualization for Sample 3: ”Person makes a move on defenders during the spring game.” HIVE captures dynamic motion cues, focusing on the athlete and defenders.

Figure 8: Gradient map visualization for Sample 4. HIVE maintains granular focus on key visual features, improving gradient flow.

Figure 9: Gradient map visualization for Sample 5. Enhanced gradient stability enables sharper feature refinement in early layers.

Figure 10: Gradient map visualization for Sample 6. HIVE consistently emphasizes meaningful visual elements across encoder layers.

Figure 11: Gradient map visualization for Sample 7. Stable gradient propagation ensures effective visual feature learning across hierarchical layers.

Figure 12: Attention map visualization for Sample 1: ”Investigators and journalists gather around the car of person after an attack on Wednesday.” HIVE emphasizes key elements such as the car and surrounding individuals, demonstrating improved semantic localization.

Figure 13: Attention map visualization for Sample 2: ”Colorful plastic and aluminum chairs leaning against tables at a cafe outdoor dining area.” HIVE highlights chairs and tables with sharper focus, enhancing object-level localization.

Figure 14: Attention map visualization for Sample 3: ”Person makes a move on defenders during the spring game.” HIVE effectively highlights the athlete’s movement and surrounding players, improving focus on dynamic elements.

Vision-Language Pretraining

Vision-language pretraining learns visual representations by relating images, video, or other visual inputs to language.

See all 80 papers

Mema: Memory-Augmented Adapter for Enhanced Vision-Language Understanding
Feb 28, 2026Ying Liu, Yudong Han, Kean Shi +1Memory-Augmented VLMs Multimodal Large Language Models
Beijing Institute of Technology · Peking University · Yangtze Delta Region Academy of Beijing Institude of Technology, Jiaxing, China
From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion
Jan 15, 2026Cheng Chen, Yuyu Guo, Pengpeng Zeng +4Vision-Language Models VLM Adaptation
Tongji University, China · Ant Group, China · Shanghai Innovation Institute, China +1
CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning
Apr 3, 2026Ankan Deria, Komal Kumar, Xilin He +4Vision-Language Models Multimodal Fusion
Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE

Feb 28, 2026J/K move · Enter open · S save

Mema: Memory-Augmented Adapter for Enhanced Vision-Language Understanding

Ying Liu, Yudong Han, Kean Shi, Liyuan Pan

Beijing Institute of Technology · Peking University · Yangtze Delta Region Academy of Beijing Institude of Technology, Jiaxing, China

Memory-Augmented VLMs Multimodal Large Language Models VLM Adaptation

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have achieved remarkable performance by aligning pretrained visual representations with the linguistic knowledge embedded in Large Language Models (LLMs). However, existing approaches typically rely on final-layer visual features or learnable multi-layer fusion, which often fail to sufficiently exploit hierarchical visual cues without explicit cross-layer interaction design. In this work, we propose a Memory-Augmented Adapter (Mema) within the vision encoder. Specifically, Mema maintains a stateful memory that accumulates hierarchical visual representations across layers, with its evolution conditioned on both query embeddings and step-wise visual features. A portion of this memory is selectively injected into token representations via a feedback mechanism, thereby mitigating the attenuation of fine-grained visual cues from shallow layers. Designed as a lightweight and plug-and-play module, Mema integrates seamlessly into pretrained vision encoders without modifying the vanilla backbone architecture. Only a minimal set of additional parameters requires training, enabling adaptive visual feature refinement while reducing training overhead. Extensive experiments across multiple benchmarks demonstrate that Mema consistently improves performance, validating its effectiveness in complex multimodal reasoning tasks. The code have been released at https://github.com/Sisiliu312/Mema.

Abstract

Vision-Language Pretraining

Explore similar work

Mema: Memory-Augmented Adapter for Enhanced Vision-Language Understanding

From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion

CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning