Large Vision-Language Models

Also known as LVLM

Large vision-language models combine a visual encoder with a language model, aligning them through feature projection, multimodal pretraining, and instruction tuning. Researchers improve their abilities with reinforcement learning, latent visual reasoning, and tools for grounding answers in image regions, especially in visual question answering, OCR, charts, and long-video understanding. To cut inference costs, methods prune visual tokens, select frames, or inspect high-resolution images selectively; other work targets hallucinations, bias, privacy, and adversarial vulnerabilities through data curation, attention interventions, and model editing. Performance is improving across many benchmarks, but reliable fine-grained spatial and temporal reasoning remains difficult, and fluent answers do not always reflect correct use of visual evidence.

Latest papers

404 papers44 papers in the last four weeks, up 63% on the four weeks before. 0.3% of all new papers.

VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models
Sep 29, 2025Ravikumar Balakrishnan, Mansi PhuteLanguage Model Steering Large Vision-Language Models
HiddenLayer, USA · Georgia Institute of Technology, USA
TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models
Sep 29, 2025Zhifang Zhang, Qiqi Tao, Jiaqi Lv +3VLM Robustness Adversarial Attacks on VLMs
School of Computer Science and Engineering, Southeast University, China · School of Electrical Engineering and Computer Science, University of Queensland, Australia · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore +2
Mitigating Cross-Image Information Leakage in Multi-Image Understanding with Large Vision-Language Models
Aug 19, 2025Yeji Park, Minyoung Lee, Sanghyuk Chun +1Large Vision-Language Models VLM Reasoning
Sogang University · Princeton University · NAVER AI Lab
SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision
Aug 5, 2025Zhaoxu Li, Chenqi Kong, Yi Yu +6VLM Evaluation LLM Hallucination Mitigation
ROSE Lab, Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · ROSE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · City University of Hong Kong, Hong Kong SAR +3
MLLM-Guided Autonomous Object Detection for Open-World Understanding
Jul 14, 2025Furkan Mumcu, Michael J. Jones, Anoop Cherian +1Open-Vocabulary Object Detection Vision-Language Grounding
University of South Florida · Mitsubishi Electric Research Laboratories (MERL)
GenRecal: Generation after Recalibration from Large to Small Vision-Language Models
Jun 18, 2025Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro +2Cross-Modal Knowledge Distillation VLM Distillation
NVIDIA · KAIST
Seeing Isn't Orienting: A Cognitively Informed Hierarchical Benchmark for Object Orientation in MLLMs
May 27, 2025Nazia Tasnim, Keanu Nichols, Yuting Yan +4VLM Evaluation Visual Spatial Reasoning
Boston University · Work done while at Boston University; currently at Johns Hopkins University
ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
May 24, 2025Duo Li, Zuhao Yang, Xiaoqin Zhang +2Efficient VLM Inference Large Vision-Language Models
CCDS, NTU, Singapore · CCST, ZJUT, China · Terminus AI Lab, UCAS, China
Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models
May 18, 2025Kai Tang, Jinhao You, Yichen Guo +8Vision-Language Models Hallucination in Language Models
Peking University · University of Pennsylvania · Nanyang Technological University +4
PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models?
Feb 6, 2025Mennatullah SiamLarge Vision-Language Models Vision-Language Grounding
Image-based Geo-localization for Robotics: Are Black-box Vision-Language Models there yet?
Jan 28, 2025Sania Waheed, Bruno Ferrarini, Michael Milford +2VLM Robustness Cross-View Geo-Localization
School of Electronics and Computer Science, University of Southampton, Southampton SO17 1BJ · MyWay srl, Via Osti, 6 - 29010 Vernasca (PC) · School of Electrical Engineering and Computer Science, Queensland University of Technology, Brisbane, QLD 4000, Australia +1
Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models
Dec 11, 2024Hee-Seon Kim, Minbeom Kim, Seokil Ham +1VLM Robustness Adversarial Attacks on VLMs
School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea
Privacy-Aware Visual Language Models
May 27, 2024Laurens Samson, Nimrod Barazani, Sennay Ghebreab +1VLM Evaluation VLM Adaptation
Socially-Intelligent Artificial Systems Group, University of Amsterdam · University of Amsterdam · Fundamental AI Lab, University of Technology Nuremberg
InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models
Dec 4, 2023Xunguang Wang, Pingchuan Ma, Zhenlan Ji +3Adversarial Attacks on VLMs Large Vision-Language Models
The Hong Kong University of Science and Technology, Hong Kong, China

Sep 29, 2025J/K move · Enter open · S save

VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models

Ravikumar Balakrishnan, Mansi Phute

HiddenLayer, USA · Georgia Institute of Technology, USA

Language Model Steering Large Vision-Language Models

Open paper AskPDF arXiv

As Vision Language Models (VLMs) are deployed across safety-critical applications, understanding and controlling their behavioral patterns has become increasingly important. Existing behavioral control methods face significant limitations: system prompting approaches could easily be overridden by user instructions, while applying activation-based steering vectors requires invasive runtime access to model internals, precluding deployment with API-based services and closed-source models. Finding steering methods that transfer across multiple VLMs is still an open area of research. To this end, we introduce universal visual input based steering for output redirection (VISOR++), to achieve behavioral control through optimized visual inputs alone. We demonstrate that a single VISOR++ image can be generated for an ensemble of VLMs to emulate each of their steering vectors. By crafting universal visual inputs that induce target activation patterns, VISOR++ eliminates the need for runtime model access while remaining deployment-agnostic. This means that when an underlying model supports multimodal capability, model behaviors can be steered by inserting an image input replacing runtime steering vector based interventions. We first demonstrate the effectiveness of the VISOR++ images on open-access models such as LLaVA-1.5-7B and IDEFICS2-8B along three alignment directions: refusal, sycophancy and survival instinct. Both the model-specific steering images and the jointly optimized images achieve performance parity closely following that of steering vectors for both positive and negative steering tasks. We also show the promise of VISOR++ images in achieving directional behavioral shifts for unseen models including both open-access and closed-access ones. Furthermore, VISOR++ images are able to preserve 99.9% performance on 14,000 unrelated MMLU evaluation tasks.