Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models

Dhruba Ghosh; Yuhui Zhang; Ludwig Schmidt

cs.CVFeb 19, 2026

Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models

Authors: Dhruba Ghosh, Yuhui Zhang, Ludwig Schmidt

Organizations: Stanford University

Abstract

Vision-language models (VLMs) have made substantial progress across a wide range of visual question answering benchmarks, spanning visual reasoning, document understanding, and multimodal dialogue. These improvements are evident in a wide range of VLMs built on a variety of base models, alignment architectures, and training data. However, recent works show that these models trail behind in traditional image classification benchmarks, which test fine-grained visual knowledge. We test a large number of recent VLMs on fine-grained classification benchmarks and identify potential factors in the disconnect between fine-grained knowledge and other vision benchmarks. Through a series of ablation experiments, we find that using a better LLM improves all benchmark scores equally, while a better vision encoder disproportionately improves fine-grained classification performance. Furthermore, we find that the pretraining stage is also vital to fine-grained performance, particularly when the language model weights are unfrozen during pretraining. These insights pave the way for enhancing fine-grained visual understanding and vision-centric capabilities in VLMs.

Figures & tables

Panel of Figure 1 — Figure 1: Overview. (Top) We investigate the fine-grained classification capabilities of vision-language models (VLMs), a crucial yet often overlooked aspect that underpins higher-level understanding and reasoning. (Bottom) Through 22 systematic ablation experiments on key model components and training strategies (left), we identify the factors that drive fine-grained classification performance in VLMs (right).

	Vision			Pretraining	Finetuning	Fine-grained	General
Model/Ablation	Encoder	LLM	Arch.	Data	Data	Classification	VQA
LLaVA-1.5-7B	CLIP L/14	Vicuna	LLaVA	LLaVA	LLaVA	59.3	41.8
No pretraining	CLIP L/14	Vicuna	LLaVA	None	LLaVA	52.8	39.3
LLaVA reproduction	CLIP L/14	Vicuna	LLaVA	LLaVA	LLaVA	57.9 (+5.1)	41.2 (+1.9)
Qwen2 LLM	CLIP L/14	Qwen2	LLaVA	LLaVA	LLaVA	63.0 (+5.1)	48.3 (+7.1)
DFN-CLIP encoder	DFN H/14	Qwen2	LLaVA	LLaVA	LLaVA	68.0 (+5.0)	49.5 (+1.2)

Table 1: Summary of important ablation settings and comparison to existing models (bolded). Upgrading the base LLM and vision encoder, as well as pretraining the connector and LLM on sufficient data, provide benefits to fine-grained performance, while instruction finetuning has a smaller effect. This is in contrast to general VQA benchmarks, where choice of LLM plays a larger part.

Figure 8: Progression of combined ablation effects. Starting with Vicuna-7B and CLIP ViT-L/14, we progressively modify the base LLM, vision encoder, and training settings to increase fine-grained classification accuracy from 52.8% to 73.4%, accounting for more than half the gap to Qwen2-VL-Chat’s 87.9%. We attribute the remaining performance difference primarily to Qwen2-VL’s extensive pretraining data of 1.4T tokens.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Vision Encoder	LLM	Arch	PT Data	FT Data	Updating	Classification	VLM
CLIP ViT-L/14@336	Vicuna-7B	LLaVA	LLaVA	LLaVA	Connector	59.3	41.8
CLIP ViT-L/14@336	Vicuna-7B	LLaVA	LLaVA	LLaVA	Connector	59.0	41.2
CLIP ViT-L/14@336	Vicuna-7B	LLaVA	None	LLaVA	Connector	52.8	39.3
CLIP ViT-L/14@336	Llama2-7B	LLaVA	None	LLaVA	Connector	52.5	41.3
DFN ViT-H/14@378	Llama2-7B	LLaVA	None	LLaVA	Connector	56.2	40.0
DFN ViT-H/14@378	Vicuna-7B	LLaVA	None	LLaVA	Connector	52.9	38.2

Appendix

Table 2: Detailed results of model ablations.

Vision Encoder	LLM	Arch	PT Data	FT Data	Updating	MMBench	MMMU	MMStar
CLIP ViT-L/14@336	Vicuna-7B	LLaVA	LLaVA	LLaVA	Connector	60.4	32.2	32.7
CLIP ViT-L/14@336	Vicuna-7B	LLaVA	LLaVA	LLaVA	Connector	58.6	32.9	32.3
CLIP ViT-L/14@336	Vicuna-7B	LLaVA	None	LLaVA	Connector	54.4	32.8	30.7
CLIP ViT-L/14@336	Llama2-7B	LLaVA	None	LLaVA	Connector	58.0	34.3	31.7
DFN ViT-H/14@378	Llama2-7B	LLaVA	None	LLaVA	Connector	53.9	34.6	31.5
DFN ViT-H/14@378	Vicuna-7B	LLaVA	None	LLaVA	Connector	51.3	33.2	30.1

Appendix

Table 3: Detailed results of model ablations on general VQA benchmarks.

Vision Encoder	LLM	Arch	PT Data	FT Data	Updating	ImageNet	Flowers	Pets	Food
CLIP ViT-L/14@336	Vicuna-7B	LLaVA	LLaVA	LLaVA	Connector	70.9	50.7	46.3	69.4
CLIP ViT-L/14@336	Vicuna-7B	LLaVA	LLaVA	LLaVA	Connector	68.7	50.4	46.5	70.3
CLIP ViT-L/14@336	Vicuna-7B	LLaVA	None	LLaVA	Connector	64.3	46.1	40.4	60.4
CLIP ViT-L/14@336	Llama2-7B	LLaVA	None	LLaVA	Connector	64.4	44.2	40.7	60.8
DFN ViT-H/14@378	Llama2-7B	LLaVA	None	LLaVA	Connector	65.4	50.5	46.2	62.8
DFN ViT-H/14@378	Vicuna-7B	LLaVA	None	LLaVA	Connector	62.3	48.7	39.2	61.3

Appendix

Table 4: Detailed results of model ablations on fine-grained classification benchmarks.

Fine-Grained Visual Perception

Fine-grained visual perception studies how models detect, distinguish, and ground subtle visual details, attributes, parts, and changes.

See all 152 papers

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
May 21, 2026Zixuan Lan, Luzhe Sun, Matthew R. Walter +1VLM Evaluation VLM Interpretability
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University
Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
Jun 17, 2026Hong-Tao Yu, Chen-Wei Xie, Yuxin Peng +2VLM Evaluation VLM Robustness
School of Computer Science and Engineering, Southeast University, China · Alibaba Group · Wangxuan Institute of Computer Technology, National Key Laboratory for Multimedia Information Processing, Peking University, China +2
Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs
Sep 10, 2025Sary Elmansoury, Islam Mesabah, Gerrit Großmann +3VLM Reasoning Image Classification
Department of Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI) · Department of Computer Science, University of Darmstadt · Department of Computer Science, University of Kaiserslautern–Landau (RPTU) +1

May 21, 2026J/K move · Enter open · S save

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?

Zixuan Lan, Luzhe Sun, Matthew R. Walter, Jiawei Zhou

University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University

VLM Evaluation VLM Interpretability Vision-Language Grounding Visually Grounded Reasoning Fine-Grained Visual Perception Visual Evidence Sensitivity in VLMs

Open paper AskPDF arXiv

Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation, answer-space expansion, and decision-level analyses beyond standard accuracy. We further complement these behavioral results with a layer-wise analysis of vision-token geometry. Throughout the experiments, we find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence that standard accuracy should have suggested. Even when the final prediction remains unchanged, the model's internal support for the correct answer may already be weakened. We further complement a representation-level analysis, which shows increasing similarity among visual tokens in deeper layers, providing a possible explanation for our findings. Together, these results suggest that current benchmarks are not sufficient to reliably evaluate fine-grained visual grounding in VLMs.

Abstract

Fine-Grained Visual Perception

Explore similar work

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?

Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis

Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs