UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking

Joan Perez; Giovanni Fusco

cs.LGMar 14, 2026

UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking

Authors: Joan Perez, Giovanni Fusco

Organizations: Urban Geo Analytics, France · Université Côte d’Azur–CNRS–AMU–Avignon Université, ESPACE, France

Abstract

Vision-Language Models (VLMs) have emerged as powerful tools for image understanding tasks, yet their practical deployment remains hindered by significant architectural heterogeneity across model families. This paper introduces UVLM (Universal Vision-Language Model Loader), a Google Colab-based framework that provides a unified interface for loading, configuring, and benchmarking multiple VLM architectures on custom image analysis tasks. UVLM currently supports two major model families -- LLaVA-NeXT and Qwen2.5-VL -- which differ fundamentally in their vision encoding, tokenization, and decoding strategies. The framework abstracts these differences behind a single inference function, enabling researchers to compare models using identical prompts and evaluation protocols. Key features include a multi-task prompt builder with support for four response types (numeric, category, boolean, text), a consensus validation mechanism based on majority voting across repeated inferences, a flexible token budget (up to 1,500 tokens) enabling users to design custom reasoning strategies through prompt engineering, and a built-in chain-of-thought reference mode for benchmarking. UVLM is designed for reproducibility, accessibility, and extensibility and as such is freely deployable on Google Colab using consumer-grade GPU resources. The paper also presents the first benchmarking of different VLMs on tasks of increasing reasoning complexity using a corpus of 120 street-view images.

Figures & tables

Figure 1: UVLM software architecture. The notebook is organized into three sequential blocks. Block 1 loads the selected VLM and configures hardware settings, Block 2 defines the analysis tasks through an interactive prompt builder and Block 3 processes all images in a folder and writes results to CSV.

Figure 2: Comparative diagram of the dual-backend inference pipelines in UVLM. Both pipelines receive the same image–prompt pair and converge at the unified response parser.

Family	Model	Params	Checkpoint ID
LLaVA-NeXT	Mistral 7B	7B	llava-hf/llava-v1.6-mistral-7b-hf
	Vicuna 7B	7B	llava-hf/llava-v1.6-vicuna-7b-hf
	Vicuna 13B	13B	llava-hf/llava-v1.6-vicuna-13b-hf
	34B	34B	llava-hf/llava-v1.6-34b-hf
	LLaMA3 8B	8B	llava-hf/llama3-llava-next-8b-hf
	72B	72B	llava-hf/llava-next-72b-hf

Table 1: Supported VLM checkpoints in UVLM.

Model	GPU	Sec/img std	Sec/img reas
Qwen2.5-VL-3B	T4	0.73	17.23
Qwen2.5-VL-7B	T4	2.17	16.23
Qwen2.5-VL-32B	A100	1.06	60.50
LLaVA Mistral 7B	T4	5.60	26.31
LLaVA Vicuna 7B	T4	3.03	17.99
LLaMA3 LLaVA 8B	L4	1.62	27.16

Table 2: Model configurations tested. GPU indicates the minimum requirement to avoid out-of-memory errors.

Figure 3: Qwen2.5-VL-32B outputs (standard and reasoning) as compared to human annotations for task 1, 2 and 4.

#	Model	N	T1 Veh.	T2 Side.	T3 Entr.	T4 Len.	T5 Veg.	Mean
1	Qwen 32B reas	120	97.7	85.0	90.4	82.5	84.2	88.0
2	Qwen 32B std	120	98.1	80.8	92.3	55.4	89.8	83.3
3	LLaVA Vic7B std	120	95.4	84.2	85.7	76.5	74.0	83.1
4	LLaVA Vic13B std	120	96.1	80.8	86.2	56.0	70.3	77.9
5	Qwen 7B reas	120	95.8	62.5	84.9	73.3	70.7	77.4
6	LLaVA Vic13B reas	120	93.6	65.8	80.7	70.4	74.5	77.0

Table 3: Overall model ranking by proximity score (%). Higher is better. † Reduced sample due to CUDA crashes on T4 (see Section 3.5 ). ‡ Partial benchmark due to prohibitive computation time.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	GPU	Load (min)	Mode	Sec/image	CC*/image	Proximity
Qwen 3B	T4	0.61	Standard	0.73	0.00033	76.0%
Qwen 3B	T4	0.61	Reasoning	17.23	0.00644	76.1%
Qwen 7B	T4	1.11	Standard	2.17	0.00090	75.4%
Qwen 7B	T4	1.11	Reasoning	16.23	0.00779	77.4%
Qwen 32B	A100	3.66	Standard	1.06	0.00519	83.3%
Qwen 32B	A100	3.66	Reasoning	60.50	0.14031	88.0%

Appendix

Table 4: Cost-performance analysis (*computing costs (CC) based on Google Colab GPU)

Model	N	MAE	Bias	Exact	$\pm$ 1	Correlation	Proximity
Qwen 3B std	60	0.38	-0.38	81.7%	91.7%	0.90	96.5%
Qwen 3B reas	118	0.43	-0.28	70.3%	92.4%	0.88	96.1%
Qwen 7B std	120	0.28	-0.21	80.8%	95.8%	0.92	97.5%
Qwen 7B reas	120	0.47	-0.43	70.8%	92.5%	0.87	95.8%
Qwen 32B std	120	0.21	-0.11	81.7%	98.3%	0.96	98.1%
Qwen 32B reas	120	0.25	-0.20	81.7%	95.0%	0.95	97.7%

Appendix

Table 5: Task 1: Motor Vehicles

Model	N	Accuracy	Sensitivity	Specificity	Cohen $\kappa$	Proximity
Qwen 3B std	60	75.0%	93.0%	29.4%	0.27	75.0%
Qwen 3B reas	118	56.8%	63.6%	21.1%	-0.11	56.8%
Qwen 7B std	120	40.0%	31.7%	84.2%	0.07	40.0%
Qwen 7B reas	120	62.5%	62.4%	63.2%	0.15	62.5%
Qwen 32B std	120	80.8%	90.1%	31.6%	0.23	80.8%
Qwen 32B reas	120	85.0%	97.0%	21.1%	0.24	85.0%

Appendix

Table 6: Task 2: Sidewalk

Model	N	MAE	Bias	Exact	$\pm$ 1	Correlation	Proximity
Qwen 3B std	60	1.35	-1.28	26.7%	61.7%	0.67	77.5%
Qwen 3B reas	118	1.16	-0.92	37.3%	72.9%	0.44	85.5%
Qwen 7B std	120	0.88	-0.67	45.0%	79.2%	0.71	89.0%
Qwen 7B reas	120	1.24	-0.59	34.2%	70.8%	0.56	84.9%
Qwen 32B std	120	0.62	-0.28	50.0%	90.8%	0.84	92.3%
Qwen 32B reas	120	0.77	-0.42	49.2%	84.2%	0.71	90.4%

Appendix

Table 7: Task 3: Pedestrian Entrances

Model	N	MAE (m)	Bias (m)	MAPE %	$\pm$ 10m	Correlation	Proximity
Qwen 3B std	60	11.4	6.1	81.8	73.3%	-0.14	70.1%
Qwen 3B reas	118	13.0	5.2	81.5	61.9%	0.08	66.5%
Qwen 7B std	120	10.5	7.7	63.8	74.2%	0.41	70.1%
Qwen 7B reas	120	9.0	-6.2	47.3	69.2%	0.29	73.3%
Qwen 32B std	120	16.5	16.0	106.1	50.0%	0.49	55.4%
Qwen 32B reas	120	5.8	0.1	35.8	85.0%	0.48	82.5%

Appendix

Table 8: Task 4: Street Frontage Length

Model	N	Exact	$\pm$ 1 class	MAE class	$\kappa$ linear	Proximity
Qwen 3B std	60	38.3%	43.3%	1.97	0.26	60.7%
Qwen 3B reas	118	37.3%	57.6%	1.23	0.31	75.4%
Qwen 7B std	120	47.5%	63.3%	0.97	0.42	80.7%
Qwen 7B reas	120	26.7%	36.7%	1.47	0.18	70.7%
Qwen 32B std	120	68.3%	80.8%	0.51	0.64	89.8%
Qwen 32B reas	120	55.8%	66.7%	0.79	0.54	84.2%

Appendix

Table 9: Task 5: Vegetation Type

Model pair	Vehicles	Sidewalk	Entrances	Length	Vegetation	Mean $\Delta$	Verdict
Qwen 3B std $\to$ Qwen 3B reas	-0.4%	-18.2%	+8.0%	-3.6%	+14.8%	+0.1%	Neutral
Qwen 7B std $\to$ Qwen 7B reas	-1.7%	+22.5%	-4.1%	+3.2%	-10.0%	+2.0%	Helps
Qwen 32B std $\to$ Qwen 32B reas	-0.4%	+4.2%	-1.9%	+27.1%	-5.7%	+4.7%	Helps
LLaVA Mis7B std $\to$ LLaVA Mis7B reas	-1.8%	+25.0%	-1.9%	-9.5%	-17.5%	-1.1%	Hurts
LLaVA Vic7B std $\to$ LLaVA Vic7B reas	-2.4%	-6.7%	-4.8%	-9.4%	-7.7%	-6.2%	Hurts
LLaMA3 8B std $\to$ LLaMA3 8B reas	-2.2%	-5.0%	-1.1%	+33.5%	+5.0%	+6.0%	Helps

Appendix

Table 10: Reasoning vs Standard: Proximity delta per task

Model	N images	Vehicles	Length	Entrances	Total NAs	NA rate
Qwen 3B std	60	0	0	0	0	0.0%
Qwen 3B reas	118	11	1	2	14	0.8%
Qwen 7B std	120	0	0	0	0	0.0%
Qwen 7B reas	120	0	0	0	0	0.0%
Qwen 32B std	120	0	0	0	0	0.0%
Qwen 32B reas	120	0	0	0	0	0.0%

Appendix

Table 11: Parsing failures (NA counts across all consensus runs)

Model	N images	Vehicles	Length	Entrances	Total	Trunc. rate
Qwen 3B std	60	0	0	0	0	0.0%
Qwen 3B reas	118	1	0	1	2	0.3%
Qwen 7B std	120	0	0	0	0	0.0%
Qwen 7B reas	120	0	3	0	3	0.5%
Qwen 32B std	120	0	0	0	0	0.0%
Qwen 32B reas	120	0	0	0	0	0.0%

Appendix

Table 12: Truncation alerts

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
Apr 17, 2026Yige Xu, Yongjie Wang, Zizhuo Wu +3Vision-Language Models VLM Adaptation
College of Computing and Data Science, Nanyang Technological University, Singapore · Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · Tongyi Lab, Alibaba Group, China
VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark
Aug 19, 2025Vy Tuong Dang, An Vo, Emilio Villa-Cueva +4VLM Evaluation Multimodal Reasoning
KAIST · MBZUAI
Distilling Visual Reasoning into Text Space
Sep 28, 2026Wenhan Yang, Nilay Naharas, Ali Payani +1CoT Distillation VLM Distillation
University of California, Los Angeles · Cisco Research

Apr 17, 2026J/K move · Enter open · S save

Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap

Yige Xu, Yongjie Wang, Zizhuo Wu, Kaisong Song+2

College of Computing and Data Science, Nanyang Technological University, Singapore · Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · Tongyi Lab, Alibaba Group, China

Vision-Language Models VLM Adaptation Visual Reasoning VLM Reasoning Modality Gap in VLMs Multimodal Model Evaluation

Open paper AskPDF arXiv

Reasoning in vision-language models (VLMs) has recently attracted significant attention due to its broad applicability across diverse downstream tasks. However, it remains unclear whether the superior performance of VLMs stems from genuine vision-grounded reasoning or relies predominantly on the reasoning capabilities of their textual backbones. To systematically measure this, we introduce CrossMath, a novel multimodal reasoning benchmark designed for controlled cross-modal comparisons. Specifically, we construct each problem in text-only, image-only, and image+text formats guaranteeing identical task-relevant information, verified by human annotators. This rigorous alignment effectively isolates modality-specific reasoning differences while eliminating confounding factors such as information mismatch. Extensive evaluation of state-of-the-art VLMs reveals a consistent phenomenon: a substantial performance gap between textual and visual reasoning. Notably, VLMs excel with text-only inputs, whereas incorporating visual data (image+text) frequently degrades performance compared to the text-only baseline. These findings indicate that current VLMs conduct reasoning primarily in the textual space, with limited genuine reliance on visual evidence. To mitigate this limitation, we curate a CrossMath training set for VLM fine-tuning. Empirical evaluations demonstrate that fine-tuning on this training set significantly boosts reasoning performance across all individual and joint modalities, while yielding robust gains on two general visual reasoning tasks. Source code is available at https://github.com/xuyige/CrossMath.

Abstract

VLM Evaluation

Explore similar work

Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap

VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark

Distilling Visual Reasoning into Text Space