Back to the Barn with LLAMAs: Evolving Pretrained LLM Backbones in Finetuning Vision Language Models

Sameera Horawalavithana; Lauren Phillips; Ian Stewart; Sai Munikoti; Karl Pazdernik

cs.AIApr 13, 2026

Back to the Barn with LLAMAs: Evolving Pretrained LLM Backbones in Finetuning Vision Language Models

Authors: Sameera Horawalavithana, Lauren Phillips, Ian Stewart, Sai Munikoti, Karl Pazdernik

Organizations: Pacific Northwest National Laboratory, Richland, WA 99352, USA

Abstract

Vision-Language Models (VLMs) have rapidly advanced by leveraging powerful pre-trained Large Language Models (LLMs) as core reasoning backbones. As new and more capable LLMs emerge with improved reasoning, instruction-following, and generalization, there is a pressing need to efficiently update existing VLMs to incorporate these advancements. However, the integration of new LLMs into VLMs, particularly how the evolving LLMs contribute to multimodal reasoning, alignment, and task-specific performance remains underexplored. Addressing this gap is important for VLM development, given the rapid evolution of pretrained LLM backbones. This study presents a controlled and systematic investigation of how changes in the pretrained LLM backbone affect downstream VLM task performance. By having the vision encoder, training data, and post-training algorithm remain same across LLAMA-1, LLAMA-2, and LLAMA-3 based VLMs, we find that newer LLM backbones do not always lead to better VLMs, but the performance depends on the downstream VLM task. For example, in visual question and answering tasks, newer LLM backbones tend to solve different questions rather than just more questions, and our analysis shows this is driven by differences in how the models process information, including better calibrated confidence and more stable internal representations. We also find that some VLM capabilities appear only in the newest LLM generation, while tasks that depend mainly on visual understanding see little benefit from a newer LLM backbone.

Figures & tables

Panel of Figure 1 — Figure 1: VLM performance with evolving pretrained LLM backbones across various domains.

Panel of Figure 4 — Figure 4: Analysis on layer-wise context vectors of the VLMs trained and tested on VQA Scene Dataset. Layer k’s context vector encodes the model state about what token to generate next, given the question and image.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model Scale / Transformer Architecture
Parameter	NeVA Default (Image)	LLAMA 3.1 Instruct (8B)	LLAMA 2 (7B)	LLAMA 1 (7B)
encoder_seq_length	4096 *	8192	4096	2048
num_layers	24 *	32	32	32
hidden_size	2048 *	4096	4096	4096
ffn_hidden_size	5440 *	14336	11008	11008
num_attention_heads	16 *	32	32	32

Appendix

Table 1: Hyperparameters across NeVA default vs. three LLAMA profiles. Identical values (BF16, TP=1, PP=1, fused_adam, etc.) are omitted.

Model	BLEU ( $\mu\pm\sigma$ )	ROUGE ( $\mu\pm\sigma$ )
LLAMA 1	$0.02046\pm 0.01791$	$0.06444\pm 0.04520$
LLAMA 2	$0.02178\pm 0.01786$	$0.06860\pm 0.04723$
LLAMA 3	$0.01967\pm 0.01800$	$0.06382\pm 0.04764$

Appendix

Table 2: VLM performance of image captioning on the SciCap test set with different LLM backbones. Metrics are reported as mean (

\mu

) and standard deviation (

\sigma

) for the best validated checkpoint of each model.

Panel of Table 4 — Table 4: Example predictions on VQA-Scene Dataset. We list the ground truth answer as what recorded by the majority of human annotators. We used the same query on all 3 model generations.

Image	Query/Ground Truth Answer	LLAMA-1 Answer	LLAMA-2 Answer	LLAMA-3 Answer
	Think about the magnetic force between the magnets in each pair. Which of the following statements is true? Context: The images below show two pairs of magnets. The magnets in different pairs do not affect each other. All the magnets shown are made of the same material, but some of them are different sizes and shapes. Options: (A) The magnitude of the magnetic force is the same in both pairs. (B) The magnitude of the magnetic force is smaller in Pair 1. (C) The magnitude of the magnetic force is smaller in Pair 2.	SOLUTION: Magnet sizes affect the magnitude of the magnetic force. Imagine magnets that are the same shape and made of the same material. The smaller the magnets, the smaller the magnitude of the magnetic force between them. Magnet A is the same size in both pairs. But Magnet B is smaller in Pair 1 than in Pair 2. So, the magnitude of the magnetic force is smaller in Pair 1 than in Pair 2. ### ANSWER: B.	SOLUTION: Magnet sizes affect the magnitude of the magnetic force. Imagine magnets that are the same shape and made of the same material. The smaller the magnets, the smaller the magnitude of the magnetic force between them. Magnet A is the same size in both pairs. But Magnet B is smaller in Pair 2 than in Pair 1. So, the magnitude of the magnetic force is smaller in Pair 2 than in Pair 1. ### ANSWER: C.	SOLUTION: Magnet sizes affect the magnitude of the magnetic force. Imagine magnets that are the same shape and made of the same material. The smaller the magnets, the smaller the magnitude of the magnetic force between them. Magnet A is the same size in both pairs. But Magnet B is smaller in Pair 1 than in Pair 2. So, the magnitude of the magnetic force is smaller in Pair 1 than in Pair 2. ### ANSWER: B.
	Is the following statement about our solar system true or false? The largest planet is made mainly of ice. Context: Use the data to answer the question below. Options: (A) true (B) false	SOLUTION: The table tells you that the largest planet is Jupiter and that Jupiter is made mainly of gas. So, the largest planet is made mainly of gas. ### ANSWER: A.	SOLUTION: To decide which planet is the largest, look at the volumes and compare the exponents. Jupiter’s volume has an exponent of 15, which is the largest of all the planets. So, the largest planet is Jupiter. Jupiter is made mainly of gas. So, the largest planet is made mainly of gas. ### ANSWER: B.	SOLUTION: The table tells you that the largest planet is Jupiter and that Jupiter is made mainly of gas. So, the largest planet is made mainly of gas. ### ANSWER: B.

Appendix

Table 5: Example predictions on ScienceQA Dataset. In the ground truth answer choices, the bold answer is the ground truth. For brevity, we omit the LECTURE block in the prediction text.

Query/Ground Truth Answer	LLAMA-1 Answer	LLAMA-2 Answer	LLAMA-3 Answer
As per the exposure map, what is the latitude and longitude of the origin (star) of the epicenter in the seismic event ”M 5.7 - 82 km NE of Pangai, Tonga” Answer: Latitude: 19.361S Longitude: 173.728W	Pangai, Neiafu, Nuku’alofa	Pangai, Nuku’alofa	Latitude: 13.764S Longitude: 172.533W
As per the exposure map, determine whether the epicenter (star) of the seismic event ”M 5.7 - 82 km NE of Pangai, Tonga” is on the land or water Answer: water	water	water	water
As per the exposure map, what is the latitude and longitude of the origin (star) of the epicenter in the seismic event ”M 5.5 - Izu Islands, Japan region” Answer: Latitude: 31.217N Longitude: 142.438E	Tokyo, Nagoya	Tokyo, Nagoya	Latitude: 32.827N Longitude: 130.754E
As per the exposure map, determine whether the epicenter (star) of the seismic event ”M 5.5 - Izu Islands, Japan region” is on the land or water Answer: water	water	water	water

Appendix

Table 6: Example predictions on Seismic Dataset.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models
Oct 27, 2026Miso Choi, Daekeun Kim, Eunji Kim +1Memory-Augmented VLMs Efficient VLM Inference
Dept. of Computer Science and Engineering, Korea University · Amazon
Deep Pre-Alignment for VLMs
May 14, 2026Tianyu Yu, Kechen Fang, Zihao Wan +5Vision-Language Models Large Vision-Language Models
1Tsinghua University · 2Taobao & Tmall Group of Alibaba · 3Shanghai Qi Zhi Institute.
Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
Feb 19, 2026Dhruba Ghosh, Yuhui Zhang, Ludwig SchmidtFine-Grained Visual Perception VLM Evaluation
Stanford University

Oct 27, 2026J/K move · Enter open · S save

V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models

Miso Choi, Daekeun Kim, Eunji Kim, Jungbeom Lee

Dept. of Computer Science and Engineering, Korea University · Amazon

Memory-Augmented VLMs Efficient VLM Inference Reasoning for VLMs

Open paper AskPDF arXiv

Embedding-level scaling expands the capacity of language models through lookup-based memory rather than by increasing model depth, width, or dense computation. Extending this idea to vision-language models (VLMs) is challenging because visual features are continuous, spatially structured, and query-dependent, unlike discrete text tokens that naturally support n-gram lookup. We introduce V-LUMEN, a visual lookup memory for VLMs. V-LUMEN discretizes visual features with a frozen codebook, builds spatially aware memory keys through spatial aggregation-based hashing, and modulates them with text-conditioned relevance signals; all addressing is non-parametric, and the retrieved embeddings are fused into the backbone through a gated residual pathway. On five reasoning-intensive benchmarks, V-LUMEN substantially improves a frozen Qwen3-VL-2B backbone and transfers zero-shot to three benchmarks whose images are disjoint from the training data. At a matched parameter budget, moving part of a LoRA or MoE adapter's capacity into V-LUMEN preserves accuracy on two backbones, indicating that external visual memory is an independent capacity axis rather than a stronger adaptation method. Because lookup cost is constant in table size and incurred only once at prefill, this capacity grows in storage rather than in per-token compute. Code is available at https://github.com/miso-choi/V-Lumen.

Abstract

Vision-Language Models

Explore similar work

V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models

Deep Pre-Alignment for VLMs

Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models