Vision-Language Models (VLMs) have rapidly advanced by leveraging powerful pre-trained Large Language Models (LLMs) as core reasoning backbones. As new and more capable LLMs emerge with improved reasoning, instruction-following, and generalization, there is a pressing need to efficiently update existing VLMs to incorporate these advancements. However, the integration of new LLMs into VLMs, particularly how the evolving LLMs contribute to multimodal reasoning, alignment, and task-specific performance remains underexplored. Addressing this gap is important for VLM development, given the rapid evolution of pretrained LLM backbones. This study presents a controlled and systematic investigation of how changes in the pretrained LLM backbone affect downstream VLM task performance. By having the vision encoder, training data, and post-training algorithm remain same across LLAMA-1, LLAMA-2, and LLAMA-3 based VLMs, we find that newer LLM backbones do not always lead to better VLMs, but the performance depends on the downstream VLM task. For example, in visual question and answering tasks, newer LLM backbones tend to solve different questions rather than just more questions, and our analysis shows this is driven by differences in how the models process information, including better calibrated confidence and more stable internal representations. We also find that some VLM capabilities appear only in the newest LLM generation, while tasks that depend mainly on visual understanding see little benefit from a newer LLM backbone.
Figures & tables
Figure 1: VLM performance with evolving pretrained LLM backbones across various domains.
Figure 2
Figure 4: Analysis on layer-wise context vectors of the VLMs trained and tested on VQA Scene Dataset. Layer k’s context vector encodes the model state about what token to generate next, given the question and image.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
NeVA Default (Image)
LLAMA 3.1 Instruct (8B)
LLAMA 2 (7B)
LLAMA 1 (7B)
Model Scale / Transformer Architecture
encoder_seq_length
4096 *
8192
4096
2048
num_layers
24 *
32
32
32
hidden_size
2048 *
4096
4096
4096
ffn_hidden_size
5440 *
14336
11008
11008
num_attention_heads
16 *
32
32
32
Appendix
Table 1: Hyperparameters across NeVA default vs. three LLAMA profiles. Identical values (BF16, TP=1, PP=1, fused_adam, etc.) are omitted.
Model
BLEU ( μ±σ )
ROUGE ( μ±σ )
LLAMA 1
0.02046±0.01791
0.06444±0.04520
LLAMA 2
0.02178±0.01786
0.06860±0.04723
LLAMA 3
0.01967±0.01800
0.06382±0.04764
Appendix
Table 2: VLM performance of image captioning on the SciCap test set with different LLM backbones. Metrics are reported as mean ( μ ) and standard deviation ( σ ) for the best validated checkpoint of each model.
Table 4: Example predictions on VQA-Scene Dataset. We list the ground truth answer as what recorded by the majority of human annotators. We used the same query on all 3 model generations.
Image
Query/Ground Truth Answer
LLAMA-1 Answer
LLAMA-2 Answer
LLAMA-3 Answer
Think about the magnetic force between the magnets in each pair. Which of the following statements is true? Context: The images below show two pairs of magnets. The magnets in different pairs do not affect each other. All the magnets shown are made of the same material, but some of them are different sizes and shapes. Options: (A) The magnitude of the magnetic force is the same in both pairs. (B) The magnitude of the magnetic force is smaller in Pair 1. (C) The magnitude of the magnetic force is smaller in Pair 2.
SOLUTION: Magnet sizes affect the magnitude of the magnetic force. Imagine magnets that are the same shape and made of the same material. The smaller the magnets, the smaller the magnitude of the magnetic force between them. Magnet A is the same size in both pairs. But Magnet B is smaller in Pair 1 than in Pair 2. So, the magnitude of the magnetic force is smaller in Pair 1 than in Pair 2. ### ANSWER: B.
SOLUTION: Magnet sizes affect the magnitude of the magnetic force. Imagine magnets that are the same shape and made of the same material. The smaller the magnets, the smaller the magnitude of the magnetic force between them. Magnet A is the same size in both pairs. But Magnet B is smaller in Pair 2 than in Pair 1. So, the magnitude of the magnetic force is smaller in Pair 2 than in Pair 1. ### ANSWER: C.
SOLUTION: Magnet sizes affect the magnitude of the magnetic force. Imagine magnets that are the same shape and made of the same material. The smaller the magnets, the smaller the magnitude of the magnetic force between them. Magnet A is the same size in both pairs. But Magnet B is smaller in Pair 1 than in Pair 2. So, the magnitude of the magnetic force is smaller in Pair 1 than in Pair 2. ### ANSWER: B.
Is the following statement about our solar system true or false? The largest planet is made mainly of ice. Context: Use the data to answer the question below. Options: (A) true (B) false
SOLUTION: The table tells you that the largest planet is Jupiter and that Jupiter is made mainly of gas. So, the largest planet is made mainly of gas. ### ANSWER: A.
SOLUTION: To decide which planet is the largest, look at the volumes and compare the exponents. Jupiter’s volume has an exponent of 15, which is the largest of all the planets. So, the largest planet is Jupiter. Jupiter is made mainly of gas. So, the largest planet is made mainly of gas. ### ANSWER: B.
SOLUTION: The table tells you that the largest planet is Jupiter and that Jupiter is made mainly of gas. So, the largest planet is made mainly of gas. ### ANSWER: B.
Appendix
Table 5: Example predictions on ScienceQA Dataset. In the ground truth answer choices, the bold answer is the ground truth. For brevity, we omit the LECTURE block in the prediction text.
Image
Query/Ground Truth Answer
LLAMA-1 Answer
LLAMA-2 Answer
LLAMA-3 Answer
As per the exposure map, what is the latitude and longitude of the origin (star) of the epicenter in the seismic event ”M 5.7 - 82 km NE of Pangai, Tonga” Answer: Latitude: 19.361S Longitude: 173.728W
Pangai, Neiafu, Nuku’alofa
Pangai, Nuku’alofa
Latitude: 13.764S Longitude: 172.533W
As per the exposure map, determine whether the epicenter (star) of the seismic event ”M 5.7 - 82 km NE of Pangai, Tonga” is on the land or water Answer: water
water
water
water
As per the exposure map, what is the latitude and longitude of the origin (star) of the epicenter in the seismic event ”M 5.5 - Izu Islands, Japan region” Answer: Latitude: 31.217N Longitude: 142.438E
Tokyo, Nagoya
Tokyo, Nagoya
Latitude: 32.827N Longitude: 130.754E
As per the exposure map, determine whether the epicenter (star) of the seismic event ”M 5.5 - Izu Islands, Japan region” is on the land or water Answer: water
Embedding-level scaling expands the capacity of language models through lookup-based memory rather than by increasing model depth, width, or dense computation. Extending this idea to vision-language models (VLMs) is challenging because visual features are continuous, spatially structured, and query-dependent, unlike discrete text tokens that naturally support n-gram lookup. We introduce V-LUMEN, a visual lookup memory for VLMs. V-LUMEN discretizes visual features with a frozen codebook, builds spatially aware memory keys through spatial aggregation-based hashing, and modulates them with text-conditioned relevance signals; all addressing is non-parametric, and the retrieved embeddings are fused into the backbone through a gated residual pathway. On five reasoning-intensive benchmarks, V-LUMEN substantially improves a frozen Qwen3-VL-2B backbone and transfers zero-shot to three benchmarks whose images are disjoint from the training data. At a matched parameter budget, moving part of a LoRA or MoE adapter's capacity into V-LUMEN preserves accuracy on two backbones, indicating that external visual memory is an independent capacity axis rather than a stronger adaptation method. Because lookup cost is constant in table size and incurred only once at prefill, this capacity grows in storage rather than in per-token compute. Code is available at https://github.com/miso-choi/V-Lumen.
Miso Choi, Daekeun Kim, Eunji Kim +1
Dept. of Computer Science and Engineering, Korea University · Amazon
Most Vision Language Models (VLMs) directly map outputs from ViT encoders to the LLM via a lightweight projector. While effective, recent analysis suggests this architecture suffers from an alignment challenge: visual features remain distant from the text space in the initial layers of the LLM, forcing the model to waste critical depth~\cite{zhang-etal-2024-investigating,artzy-schwartz-2024-attend} on superficial modality alignment rather than deep understanding and complex reasoning. In this work, we propose Deep Pre-Alignment (DPA), a novel architecture that replaces the standard ViT encoder with a small VLM as perceiver, ensuring visual features are deeply aligned with the text space of the target large language model. Comprehensive experiments demonstrate the effectiveness of DPA. On the 4B parameter scale, DPA outperforms baselines by 1.9 points across 8 multimodal benchmarks, with gains widening to 3.0 points at the 32B scale. Moreover, by offloading alignment to the perceiver, DPA achieves a 32.9% reduction in language capability forgetting over 3 text benchmarks. We further demonstrate that these gains are consistent across different LLM families including Qwen3 and LLaMA 3.2, highlighting the generality of our approach. Beyond performance, DPA also offers a seamless upgrade path for current VLM development, requiring only a modular replacement for the visual encoder with marginal computation overhead.
Tianyu Yu, Kechen Fang, Zihao Wan +5
1Tsinghua University · 2Taobao & Tmall Group of Alibaba · 3Shanghai Qi Zhi Institute.
Vision-language models (VLMs) have made substantial progress across a wide range of visual question answering benchmarks, spanning visual reasoning, document understanding, and multimodal dialogue. These improvements are evident in a wide range of VLMs built on a variety of base models, alignment architectures, and training data. However, recent works show that these models trail behind in traditional image classification benchmarks, which test fine-grained visual knowledge. We test a large number of recent VLMs on fine-grained classification benchmarks and identify potential factors in the disconnect between fine-grained knowledge and other vision benchmarks. Through a series of ablation experiments, we find that using a better LLM improves all benchmark scores equally, while a better vision encoder disproportionately improves fine-grained classification performance. Furthermore, we find that the pretraining stage is also vital to fine-grained performance, particularly when the language model weights are unfrozen during pretraining. These insights pave the way for enhancing fine-grained visual understanding and vision-centric capabilities in VLMs.