cs.CVMar 6, 2026

Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders

Authors: Boqiang Zhang, Lei Ke, Ruihan Yang, Qi Gao, Tianyuan Qu, Rossell Chen, Dong Yu, Leoweiliang

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision Language Model (VLM) development has largely relied on scaling model size, which hinders deployment on compute-constrained mobile and edge devices such as smartphones and robots. In this work, we explore the performance limits of compact (e.g., 2B and 8B) VLMs. We challenge the prevailing practice that state-of-the-art VLMs must rely on vision encoders initialized via massive contrastive pretraining (e.g., CLIP/SigLIP). We identify an objective mismatch: contrastive learning, optimized for discrimination, enforces coarse and category-level invariances that suppress fine-grained visual cues needed for dense captioning and complex VLM reasoning. To address this issue, we present Penguin-VL, whose vision encoder is initialized from a text-only LLM. Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining, unlocking a higher degree of visual fidelity and data efficiency for multimodal understanding. Across various image and video benchmarks, Penguin-VL achieves performance comparable to leading VLMs (e.g., Qwen3-VL) in mathematical reasoning and surpasses them in tasks such as document understanding, visual knowledge, and multi-perspective video understanding. Notably, these gains are achieved with a lightweight architecture, demonstrating that improved visual representation rather than model scaling is the primary driver of performance. Our ablations show that Penguin-Encoder consistently outperforms contrastive-pretrained encoders, preserving fine-grained spatial and temporal cues that are critical for dense perception and complex reasoning. This makes it a strong drop-in alternative for compute-efficient VLMs and enables high performance in resource-constrained settings. Code: https://github.com/tencent-ailab/Penguin-VL

Figures & tables

Figure 1: Benchmark comparison across tasks. Left (Image): Image benchmarks grouped by capability: OCR , Math , and Knowledge . Right (Video): Video benchmarks grouped into Long-form & Temporal and General Understanding . Each sector corresponds to a benchmark, and the radial bars indicate relative performance, where longer bars mean better results. Overall, Penguin-VL 2B achieves excellent performance across all modalities, highlighting clear advantages over existing state-of-the-art opensource models.
Figure 2: Comparison of Vision Encoder Training Paradigms for VLMs. . We contrast three distinct approaches for training the vision encoder: (a) Contrastive Training , which relies on contrastive loss between image and text encoders and then frozen the vision encoder during VLM training. While the training logic is simple, this paradigm requires massive datasets, suffers from training instability, and often yields insufficient fine-grained multimodal alignment. (b) Direct LLM Supervision initializes the vision encoder with contrastive pre-trained weights and aligns visual features to a frozen LLM (via language modeling loss). While this enables direct alignment with the LLM feature space, it is highly sensitive to data quality and prone to overfitting to the training image distribution. (c) Penguin-Encoder Training (Ours) , which fuses the advantages of previous methods by initializing the vision encoder directly with the weights of a Text LLM. This approach ensures the starting model distributions are close—facilitating easier alignment—while equipping the vision model with rich initial linguistic knowledge and enabling simple, efficient scaling of vision parameters.
Figure 3: Method Overview. Penguin-VL adopts a unified architecture for vision understanding. Vision: The vision encoder is initialized from a text-only LLM (Qwen3-0.6B) and equiped with 2D-RoPE and bidirectional attention. To handle long video contexts efficiently, we employ a Temporal Redundancy-Aware (TRA) strategy that dynamically allocates token budgets among key frames and intermediate frames.
Figure 4: Multi-granularity video annotation. This figure illustrates the alignment between visual content and textual descriptions across three temporal scales: Dense time-level, Paragraph-level, and Video-level. Semantic entities are color-coded to distinguish Subjects , Actions , Objects & Details , Spatial Context , Mood & Tone , and Knowledge & Text .
Figure 5: Data formats for different data types. ❶ For image sequence, we use "\n" to separate image tokens from different image; ❷ For video sequence, we use "Time: xxs" to indicate timestamps of each frame, "," to separate different frames, and "\n" to separate tokens from different videos; ❸ For streaming video sequence, videos and texts are organized in an interleaved format.
Figure 6: Data mixture used in the pre-training stage. Up: Overall composition of the pre-training data samples, including general caption data and other data sources. Bottom: Detailed breakdown of the other data, covering a diverse set of domain-specific datasets.
Figure 7: Image data composition of the SFT stage.
Figure 8: Video data composition of the SFT stage. The datasets are categorized into four domains: General video understanding, action recognition & reasoning, temporal grounding & reasoning, and ego video understanding, with specific sampling ratios indicated in parentheses for each data source.
Table 1: Results comparison for 2B model variants. Best-performing results among all compared models are shown in bold , while the second-best results are underlined .
Table 2: Results comparison for 8B model variants. Best-performing results among all compared models are shown in bold , while the second-best results are underlined .
 Training Stages and ConfigurationEvaluation Results
Vision EncoderStage 0 Contrastive LearningStage 1 Encoder PretrainingStage 2 Modality AlignmentStage 3 SFTTotal data sizeAvgAI2DMathVistaChartQAMMMU-ProRealworld QA
Penguin-Encoder Pretraining Ablation
Penguin-Encoder ( using random init)     31.357.222.012.418.846.0
Penguin-Encoder ( w/o reconstruction)     32.655.629.911.618.946.9
Penguin-Encoder ( w/o relation loss)     33.356.329.517.418.245.0
Penguin-Encoder-24MDenseFusion -1MLLaVA- 665k∼\sim 24M34.656.429.017.419.351.0
Table 3: Ablation of Penguin-Encoder and comparison for LMM integration. Results are reported on five benchmarks with the average score. We categorize the training pipelines into four stages, and list approximate data sample scales for each stage. Qwen3VL-ViT and VL3-NaViT are obtained via continued pre-training on large-scale contrastively pre-trained SigLIP models, with Stage 1 data scales taken from their original papers. SigLIP2 (original resolution) follows the official preprocessing, while SigLIP2 (any resolution) adopts NaViT-style dynamic resolution with relative positional encoding. All SigLIP2 variants use siglip2-so400m-patch16-naflex. *: This value is our estimate based on the Qwen3-VL report, which only provides the total number of tokens.
Figure 9: Case study of logical reasoning and algorithm code writing. The input figure caption is for illustration only and is not provided to the model.
Figure 10: Case study of OCR and dense document understanding. The input figure caption is for illustration only.
Figure 11: Case study of chart interpretation and understanding. The input figure caption is for illustration only and is not provided to the model.
Figure 12: Case study of artistic interpretation and poetic creative writing. The input figure caption is for illustration only and is not provided to the model.
Figure 13: Case study of comprehensive video understanding and temporal grounding. The input figure caption is for illustration only.
Figure 14: Case study of short video understanding and temporal grounding. The input figure caption is for illustration only and is not provided to the model.
More in this topic

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 722 papers

Explore similar work

CardsList
Apr 3, 2026

CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning

Recent vision-language models (VLMs) typically rely on a single vision encoder trained with contrastive image-text objectives, such as CLIP-style pretraining. While contrastive encoders are effective for cross-modal alignment and retrieval, self-supervised visual encoders often capture richer dense semantics and exhibit stronger robustness on recognition and understanding tasks. In this work, we investigate how to scale the fusion of these complementary visual representations for vision-language modeling. We propose CoME-VL: Complementary Multi-Encoder Vision-Language, a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder. Our approach performs representation-level fusion by (i) entropy-guided multi-layer aggregation with orthogonality-constrained projections to reduce redundancy, and (ii) RoPE-enhanced cross-attention to align heterogeneous token grids and produce compact fused visual tokens. The fused tokens can be injected into a decoder-only LLM with minimal changes to standard VLM pipelines. Extensive experiments across diverse vision-language benchmarks demonstrate that CoME-VL consistently outperforms single-encoder baselines. In particular, we observe an average improvement of 4.9% on visual understanding tasks and 5.4% on grounding tasks. Our method achieves state-of-the-art performance on RefCOCO for detection while improving over the baseline by a large margin. Finally, we conduct ablation studies on layer merging, non-redundant feature mixing, and fusion capacity to evaluate how complementary contrastive and self-supervised signals affect VLM performance.

Ankan Deria, Komal Kumar, Xilin He +4

Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE

Vision-Language ModelsMultimodal FusionContrastive Learning for VLMs
AskPDF
Jun 3, 2026

Stateful Visual Encoders for Vision-Language Models

Vision-language models (VLMs) are increasingly used in multi-image, multi-turn agentic settings where decisions depend on visual changes. However, in existing open-weight VLMs, visual comparisons happen only inside the language model, while the visual encoder itself remains stateless: each image is encoded independently, without access to the prior visual context. As a result, small but task-critical changes may be attenuated before the language model has a chance to compare them, especially when those changes do not affect the high-level semantics of the scene. We introduce a Stateful Visual Encoder, which conditions each visual representation on prior visual features. Under supervised finetuning, VLMs equipped with stateful encoders achieve consistent improvements on controlled tasks involving cross-image spatial aggregation, multi-object visual differencing, and visual trajectory behavior cloning. These improvements are consistent across input resolutions, language model sizes, and VLM backbones. Finally, we validate our model on real-world tasks, including longitudinal radiology, fine-grained image comparison, and remote sensing, where stateful encoders consistently improve generalist VLM baselines and can match or surpass specialized models in selected domains. Project page: https://statefulvisualencoders.github.io/

Zirui Wang, Junwei Yu, Adam Yala +3

1Voio, Inc. · 2UC Berkeley · 3UCSF

Vision-Language ModelsVisual Representation LearningVLM AdaptationTemporal Representation Learning
AskPDF
May 14, 2026

Deep Pre-Alignment for VLMs

Most Vision Language Models (VLMs) directly map outputs from ViT encoders to the LLM via a lightweight projector. While effective, recent analysis suggests this architecture suffers from an alignment challenge: visual features remain distant from the text space in the initial layers of the LLM, forcing the model to waste critical depth~\cite{zhang-etal-2024-investigating,artzy-schwartz-2024-attend} on superficial modality alignment rather than deep understanding and complex reasoning. In this work, we propose Deep Pre-Alignment (DPA), a novel architecture that replaces the standard ViT encoder with a small VLM as perceiver, ensuring visual features are deeply aligned with the text space of the target large language model. Comprehensive experiments demonstrate the effectiveness of DPA. On the 4B parameter scale, DPA outperforms baselines by 1.9 points across 8 multimodal benchmarks, with gains widening to 3.0 points at the 32B scale. Moreover, by offloading alignment to the perceiver, DPA achieves a 32.9% reduction in language capability forgetting over 3 text benchmarks. We further demonstrate that these gains are consistent across different LLM families including Qwen3 and LLaMA 3.2, highlighting the generality of our approach. Beyond performance, DPA also offers a seamless upgrade path for current VLM development, requiring only a modular replacement for the visual encoder with marginal computation overhead.

Tianyu Yu, Kechen Fang, Zihao Wan +5

1Tsinghua University · 2Taobao & Tmall Group of Alibaba · 3Shanghai Qi Zhi Institute.

Vision-Language ModelsLarge Vision-Language ModelsVision-Language Alignment
AskPDF