Large Vision-Language Models

Also known as LVLM

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 36% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 248

All topics

TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models
Sep 29, 2025Zhifang Zhang, Qiqi Tao, Jiaqi Lv +3VLM Robustness Adversarial Attacks on VLMs
School of Computer Science and Engineering, Southeast University, China · School of Electrical Engineering and Computer Science, University of Queensland, Australia · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore +2
Mitigating Cross-Image Information Leakage in Multi-Image Understanding with Large Vision-Language Models
Aug 19, 2025Yeji Park, Minyoung Lee, Sanghyuk Chun +1Large Vision-Language Models VLM Reasoning
Sogang University · Princeton University · NAVER AI Lab
SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision
Aug 5, 2025Zhaoxu Li, Chenqi Kong, Yi Yu +6VLM Evaluation LLM Hallucination Mitigation
ROSE Lab, Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · ROSE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · City University of Hong Kong, Hong Kong SAR +3
GenRecal: Generation after Recalibration from Large to Small Vision-Language Models
Jun 18, 2025Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro +2Cross-Modal Knowledge Distillation VLM Distillation
NVIDIA · KAIST
Seeing Isn't Orienting: A Cognitively Informed Hierarchical Benchmark for Object Orientation in MLLMs
May 27, 2025Nazia Tasnim, Keanu Nichols, Yuting Yan +4VLM Evaluation Visual Spatial Reasoning
Boston University · Work done while at Boston University; currently at Johns Hopkins University
ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
May 24, 2025Duo Li, Zuhao Yang, Xiaoqin Zhang +2Efficient VLM Inference Large Vision-Language Models
CCDS, NTU, Singapore · CCST, ZJUT, China · Terminus AI Lab, UCAS, China
Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models
May 18, 2025Kai Tang, Jinhao You, Yichen Guo +8Vision-Language Models Hallucination in Language Models
Peking University · University of Pennsylvania · Nanyang Technological University +4
Image-based Geo-localization for Robotics: Are Black-box Vision-Language Models there yet?
Jan 28, 2025Sania Waheed, Bruno Ferrarini, Michael Milford +2VLM Robustness Cross-View Geo-Localization
School of Electronics and Computer Science, University of Southampton, Southampton SO17 1BJ · MyWay srl, Via Osti, 6 - 29010 Vernasca (PC) · School of Electrical Engineering and Computer Science, Queensland University of Technology, Brisbane, QLD 4000, Australia +1
Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models
Dec 11, 2024Hee-Seon Kim, Minbeom Kim, Seokil Ham +1VLM Robustness Adversarial Attacks on VLMs
School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea
Privacy-Aware Visual Language Models
May 27, 2024Laurens Samson, Nimrod Barazani, Sennay Ghebreab +1VLM Evaluation VLM Adaptation
Socially-Intelligent Artificial Systems Group, University of Amsterdam · University of Amsterdam · Fundamental AI Lab, University of Technology Nuremberg
InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models
Dec 4, 2023Xunguang Wang, Pingchuan Ma, Zhenlan Ji +3Adversarial Attacks on VLMs Large Vision-Language Models
The Hong Kong University of Science and Technology, Hong Kong, China
AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruning
Date pendingKyuan Oh, Bumsoo KimEfficient VLM Inference Large Vision-Language Models
Chung-Ang University, Seoul, Korea
Geometric Coastline Localization using Vision-Language Models
Date pendingRafia Malik, Bernhard Pfahringer, Karin Bryan +2Remote Sensing Image Understanding Geospatial Foundation Models
The University of Waikato, New Zealand · The University of Auckland, New Zealand

Sep 29, 2025, cs.CVJ/K move · Enter open · S save

TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models

Zhifang Zhang, Qiqi Tao, Jiaqi Lv, Na Zhao+2

School of Computer Science and Engineering, Southeast University, China · School of Electrical Engineering and Computer Science, University of Queensland, Australia · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore+2

VLM Robustness Adversarial Attacks on VLMs Vision-Language Models Large Vision-Language Models Backdoor Attacks on VLMs

Open paper AskPDF arXiv

Large vision-language models (LVLMs) have achieved impressive performance across a wide range of vision-language tasks, while they remain vulnerable to backdoor attacks. Existing backdoor attacks on LVLMs aim to force the victim model to generate a predefined target pattern, which is either inserted into or replaces the original content. We find that these fixed-pattern attacks are relatively easy to detect, because the attacked LVLM tends to memorize such frequent patterns in the training dataset, thereby exhibiting overconfidence on these targets given poisoned inputs. To address these limitations, we introduce TokenSwap, a more evasive and stealthy backdoor attack that focuses on the compositional understanding capabilities of LVLMs. Instead of enforcing a fixed targeted content, TokenSwap subtly disrupts the understanding of object relationships in text. Specifically, it causes the backdoored model to generate outputs that mention the correct objects in the image but misrepresent their relationships (i.e., bags-of-words behavior). During training, TokenSwap injects a visual trigger into selected samples and simultaneously swaps the grammatical roles of key tokens in the corresponding textual answers. However, the poisoned samples exhibit only subtle differences from the original ones, making it challenging for the model to learn the backdoor behavior. To address this, TokenSwap employs an adaptive token-weighted loss that explicitly emphasizes the learning of swapped tokens, such that the visual triggers and bags-of-words behavior are associated. Extensive experiments demonstrate that TokenSwap achieves high attack success rates while maintaining superior evasiveness and stealthiness across multiple benchmarks and various LVLM architectures.

Momentum

Related topics

Latest papers 248