Empirical Recipes for Efficient and Compact Vision-Language Models

Jiabo Huang; Zhizhong Li; Sina Sajadmanesh; Weiming Zhuang; Lingjuan Lyu

cs.CVMar 17, 2026

Empirical Recipes for Efficient and Compact Vision-Language Models

Authors: Jiabo Huang, Zhizhong Li, Sina Sajadmanesh, Weiming Zhuang, Lingjuan Lyu

Organizations: Sony AI.

Abstract

Deploying vision-language models (VLMs) in resource-constrained settings demands low latency and high throughput, yet existing compact VLMs often fall short of the inference speedups their smaller parameter counts suggest. To explain this discrepancy, we conduct an empirical end-to-end efficiency analysis and systematically profile inference to identify the dominant bottlenecks. Based on these findings, we develop optimization recipes tailored to compact VLMs that substantially reduce latency while preserving accuracy. These techniques cut time to first token (TTFT) by 53% on InternVL3-2B and by 93% on SmolVLM-256M. Our recipes are broadly applicable across both VLM architectures and common serving frameworks, providing practical guidance for building efficient VLM systems. Beyond efficiency, we study how to extend compact VLMs with structured perception outputs and introduce the resulting model family, ArgusVLM. Across diverse benchmarks, ArgusVLM achieves strong performance while maintaining a compact and efficient design.

Figures & tables

Figure 1: ArgusVLM excels at both performance and inference efficiency with a faster time-to-first-token (TTFT) compared to existing compact VLMs. Each bubble represents a model variant, where the area indicates model size.

Figure 2: ArgusVLM -2B achieves strong performance on image understanding and captioning tasks across five benchmarks compared with leading vision-language models such as QwenVL ( Bai et al., 2025b ) and InternVL ( Zhu et al., 2025 ) .

Figure 3: Flame graph from profiling VLM inference with austin ( Tornetta, 2026 ) to identify CPU-side bottlenecks. The visualization separates processes for easier analysis: Process 2953917 performs multimodal preprocessing, while Process 2954325 runs model inference. Best viewed digitally with zoom.

Panel of Figure 4 — Figure 4: GPU profiling of SmolVLM-256M inference under vLLM with (a) bfloat16 and (b) W8A8 quantization. The timelines show GPU kernels within a transformer layer during decoding, with the average execution time ( $\mu$ s) annotated below each kernel. Despite reduced precision, the quantized model is slower.

Panel of Table 1 — Table 1: Impact of incremental optimizations on InternVL3-2B inference. The first row is baseline performance under vLLM.

Panel of Table 2 — Table 2: Impact of incremental optimizations on SmolVLM-256M inference. The first row is baseline performance under vLLM.

Figure 5: Overview of the ArgusVLM architecture. ArgusVLM consists of a Vision Transformer (ViT), a single-layer MLP, and a large language model (LLM). Given an input image, we first resize it to the closest aspect ratio from a predefined set, then split it into square tiles that are resized to the ViT input resolution; we also generate a global thumbnail to provide scene-level context. Each tile (and the thumbnail) is independently encoded by the ViT into patch tokens, which are then compressed by concatenating features from spatially adjacent patches along the channel dimension. In parallel, the user instruction is tokenized into text tokens. Finally, the visual and text tokens are concatenated and fed to the LLM, which generates the response autoregressively.

Model	#Parameters	VQA			Captioning
		GQA	POPE	VQAv2	COCO2017		NoCaps
		Exact Match ↑	Accuracy ↑	Accuracy ↑	BLEU4 ↑	CIDEr ↑	BLEU4 ↑	CIDEr ↑
SmolVLM* ( Marafioti et al., 2025 )	256M	41.5	80.1	66.0	0.056	0.118	0.087	0.182
ArgusVLM (Ours)	256M	55.3	87.3	70.4	0.347	1.191	0.409	1.014
SmolVLM* ( Marafioti et al., 2025 )	500M	45.3	85.9	70.4	0.184	0.712	0.273	0.722
ArgusVLM (Ours)	500M	58.2	86.3	74.1	0.369	1.263	0.438	1.086

Table 3: Comparison to the leading vision-language models on text-based image understanding benchmarks. Results of models marked with * are measured by their open-source checkpoints provided on Huggingface ( Wolf et al., 2019 ) using LMMs-Eval ( Zhang et al., 2024 ) .

Panel of Figure 6 — Figure 6: Efficiency comparison between ArgusVLM and leading VLMs. ArgusVLM outperforms both SmolVLM and InternVL3 on TTFT and throughput. As Qwen3-VL-2B does not use tiling, we compare it to ArgusVLM -2B under the patch=1 setting.

Model	Bbox Format	mAP (%)
ArgusVLM -256M	Plain-text	1.89
ArgusVLM -256M	Location Tokens	1.98
Florence2-base (0.23B)	Location Tokens	1.67
ArgusVLM -500M	Plain-text	2.18
ArgusVLM -500M	Location Tokens	3.21
Florence2-large (0.77B)	Location Tokens	1.48

Table 4: Comparison of bounding-box representations for dense captioning on Visual Genome.

Panel of Figure 7 — Figure 7: Qualitative results from ArgusVLM -2B. Top: VQA and image captioning. Bottom: dense captioning.

Explore similar work

CIVIC: End-to-End Sequence Compactness for Efficient Vision-Language Models
May 27, 2026Fengze Yang, Bo Yu, Xuewen Luo +2Efficient VLM Inference KV-Cache Compression
Department of Civil & Environmental Engineering University of Utah 201 Presidents’ Cir, Salt Lake City, UT 84112, USA
Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
Apr 7, 2026Jun Zhang, Yicheng Ji, Feiyang Ren +7Efficient VLM Inference Large Vision-Language Models
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
Mar 30, 2026Surendra Pathak, Bo HanEfficient VLM Inference Visual Token Compression
Computer Science Department, George Mason University, Fairfax, VA - 22030, USA

May 27, 2026J/K move · Enter open · S save

CIVIC: End-to-End Sequence Compactness for Efficient Vision-Language Models

Fengze Yang, Bo Yu, Xuewen Luo, Cathy Liu+1

Department of Civil & Environmental Engineering University of Utah 201 Presidents’ Cir, Salt Lake City, UT 84112, USA

Efficient VLM Inference KV-Cache Compression Visual Token Pruning Visual Token Compression

Open paper AskPDF arXiv

Vision-Language Models (VLMs) face severe memory and latency bottlenecks due to high-resolution visual tokens. While current token reduction methods theoretically save FLOPs, post-hoc pruning introduces structural overhead, failing to yield proportional wall-clock acceleration. However, enforcing a contiguous compact pathway risks geometric disorientation and loss of fine-grained localization. To overcome these barriers, this paper introduces CIVIC, a path-consistent compact visual inference framework. By maintaining compact sequence representations seamlessly across the vision encoder, projection layer, LLM prefill, and KV-cache, CIVIC avoids non-contiguous memory access and localized unmerging overheads. Evaluated on the Qwen3-VL architecture, CIVIC successfully translates sequence reductions into genuine physical hardware efficiency, shrinking KV-cache memory to approximately one-third of the baseline and reducing end-to-end inference latency. Enabled by text-aligned KL distillation and an adaptive spatial retention floor, CIVIC achieves these efficiency milestones without degrading accuracy across rigorous multimodal reasoning and visual grounding benchmarks.

Abstract

Explore similar work

CIVIC: End-to-End Sequence Compactness for Efficient Vision-Language Models

Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects

Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies