VLM Adaptation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

34 papers in the last four weeks, up 209% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 395

All topics

One Model, Many Graphs: Learning over Attributed Graphs across Heterogeneous Modalities with Vision-Language Models
Jul 21, 2026Jiayi Yang, Yifang Chen, Yuanfu Sun +2Graph Representation Learning VLM Adaptation
New York University Shanghai · New York University
KALE: Kernel Alignment with Loss Equilibration for Stable CLIP-DINOv2 Alignment at Web Scale
Jul 21, 2026Michał PawłowiczVLM Adaptation Adaptive Loss Weighting
Open-Vocabulary Gaze Object Prediction: Benchmark and Method
Jul 21, 2026Binglu Wang, Sensen Niu, Ying Chen +1VLM Adaptation
Xi’an University of Architecture and Technology · College of Artificial Intelligence and Rebotics · Xi’an, China +3
Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio
Jul 21, 2026Abdul Basit Tonmoy, Kazi Fardinul Hoque, Md. Shahrier Islam Arham +1VLM Adaptation Multimodal Embedding
1Eximius Labs · 2Wabash College · 3Skop Intelligence Co.
PRiSM: Prototype Regularization for Few-Shot VLMs
Jul 20, 2026Ghassen Baklouti, Omprakash Chakraborty, Jose Dolz +1VLM Adaptation Prototype Learning
ÉTS Montreal
PGN: Design and Implementation of a Vision-Language Navigation System Based on Pangu Multimodal Foundation Model
Jul 20, 2026Li Xian, Mingxi Li, Yizheng Wang +3VLM Adaptation Multimodal Large Language Models
School of Information and Communication Engineering University of Electronic Science and Technology of China
Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation
Jul 20, 2026Lingrui Li, Nan Pu, Dong Zhao +4Memory-Augmented VLMs VLM Adaptation
School of Computer Science, University of Nottingham, UK · School of Computer Science and Information Engineering, Hefei University of Technology, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore
DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation
Jul 20, 2026Siobhan Reid, Zhixiang Chi, Li Gu +3Test-Time Adaptation Few-Shot Domain Adaptation
Concordia University, Montreal QC H3G 1M8, Canada · University of Toronto, Toronto ON M5S 1A1, Canada · Mila – Quebec AI Institute, Montreal QC H2S 3H1, Canada
Can Experts Adapt Without Training? On Test-Time Modality Generalization in MVLMs
Jul 18, 2026Raza Imam, Darakshan Rashid, Yutong Xie +3VLM Robustness VLM Adaptation
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Indian Institute of Technology Delhi, India · Khalifa University, Abu Dhabi, UAE
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
Jul 17, 2026Jiazhen Huang, Zhiming Liu, Changhu Wang +3Class-Imbalanced Learning Test-Time Adaptation
Tsinghua University, China · Fred Hutchinson Cancer Center, USA · Peking University, China +2
U-shaped Multi-granularity Learning for Vision-Language Models
Jul 16, 2026Biao Chen, Yunqian Yu, Xiangxu Zhao +3VLM Adaptation Visual Prompt Learning
University of Electronic Science and Technology of China · Hithink Research, Hangzhou, China
Action QFormer: Structured Representation Shaping under Action Supervision in Vision-Language-Action Models
Jul 16, 2026Yufeng Ji, Wenhao Tang, Haoyi Niu +3VLM Adaptation Vision-Language-Action Models
Shanghai Qizhi Institute · The Chinese University of Hong Kong · Hong Kong Embodied AI Lab +2
Fine-grained CLIP fine-tuning with self-annotated region alignment
Jul 15, 2026Chenyang Zhao, Wei Lin, Antoni B. Chan +1VLM Adaptation Vision-Language Grounding
Department of Computer Science, City University of Hong Kong · Division of Social Science, Hong Kong University of Science & Technology
MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuning
Jul 14, 2026Yumiao Zhao, Bo Jiang, Min Lu +2VLM Adaptation Quantum Machine Learning
Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei, China · Origin Quantum Computing Company Limited, Hefei, Anhui 230026, China
PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs
Jul 13, 2026Jing Liu, Kun Yang, Yan Wang +5Multimodal Federated Learning Federated Learning Aggregation
College of Future Information Technology, Fudan University, Shanghai 200433, China · Division of Natural and Applied Sciences, Duke Kunshan University, Suzhou 215316, China · Department of Electrical and Computer Engineering, The University of British Columbia, BC V6T 1Z4, Canada +8
MED-DSLC: Multi-Expert-Domain Classification via Domain Supervision and Logit Calibration
Jul 13, 2026Zheng Zeng, Deepak Sridhar, Nuno VasconcelosVLM Robustness VLM Adaptation
University of California, San Diego, USA
Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos
Jul 12, 2026Hao Zheng, Jinyi Huang, Tiantian Zheng +2VLM Adaptation Video Action Recognition
Department of Computer Engineering, New York University Abu Dhabi, UAE · Department of Mechanical and Mechatronics Engineering, The University of Auckland, New Zealand · Center for Quantum and Topological Systems, New York University Abu Dhabi, UAE +1
Traj-VLN: Learning Pixel-Space Interaction via Autoregressive Trajectory Generation
Jul 12, 2026Changfei Fu, Guangcheng Chen, Wenjun Xu +1VLM Adaptation Vision-Language Navigation
Answer-Conditioned Chain-of-Thought Distillation for Few-Shot Industrial Vision with Small VLMs
Jul 12, 2026Shubham RaoCoT Distillation VLM Adaptation
Entropy AI Research Labs Private Limited
Generalize LMMs to Versatile Visual Modalities via Fabricated Modality Synthesis
Jul 11, 2026Shihao Yuan, Yuanze Li, Ruyi Zhang +2VLM Adaptation Multimodal Large Language Models
Faculty of Computing, Harbin Institute of Technology, Harbin, China
TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models
Jul 10, 2026Tianyou Jiang, Ziyu ZhouFew-Shot Domain Adaptation VLM Adaptation
University of Bern, Bern, Switzerland · Shanghai Jiao Tong University, Shanghai, China
Decoupling Language Guidance from Backbones for Text-Guided Medical Segmentation
Jul 10, 2026Yungeng Liu, Xuanzi Fang, Haijin Zeng +2VLM Adaptation Text-Guided Image Segmentation
Harbin Institute of Technology (Shenzhen) · Shenzhen, China · NingBo No.2 Hospital +1
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Jul 10, 2026Xingyu Zhu, Huanshen Wu, Shuo Wang +4VLM Robustness VLM Adaptation
National University of Singapore · University of Science and Technology of China · The Hong Kong University of Science and Technology
Parameter-Efficient Vision-Language Adaptation with Continuous Metadata Conditioning for Animal Re-Identification
Jul 10, 2026Anil Osman Tur, Tonje Knutsen Sordalen, Kim Tallaksen Halvorsen +1Animal Re-Identification Distribution Shift
Department of Computer Science, University of Verona, Verona, 37134, Italy · Institute of Marine Research, Nye Flødevigveien 20, 4817 His, Norway · University of Agder, Centre for Coastal Research, Grimstad, 4604, Norway
CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding
Jul 9, 2026Yuri Ishitoya, Jeremy Siburian, Masashi Hamaya +3VLM Adaptation Robotic Manipulation
Ochanomizu University · OMRON SINIC X Corporation · University of Tokyo +1
When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs
Jul 8, 2026Tanay Sodha, Aditya Sharma, Ramya Hebbalaguppe +2Vision-Language Models VLM Adaptation
Department of Computer Science and Information Systems, Birla Institute of Technology and Science, Pilani, India · TCS Research, New Delhi, India
Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering
Jul 8, 2026Miguel Lopez-Duran, Elena Marrero, Julian Fierrez +8VLM Evaluation Document Understanding
BiometricsAI, Universidad Autónoma de Madrid (UAM), Spain
GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model
Jul 8, 2026Peter Bohm, Saimunur Rahman, Abdelwahed Khamis +3VLM Adaptation Robot Navigation
CSIRO Technology, Australia
MonoIR-RS: Infrared Remote Sensing Vision-Language Learning with CLIP and VLM Adaptation
Jul 7, 2026Jiaju Han, Ma Yaqi, Yahui Chai +8Remote Sensing Image Understanding VLM Adaptation
China University of Petroleum-Beijing at Karamay, Karamay, Xinjiang, China · Guizhou University, Guiyang, China · Shenzhen Research Institute of Big Data, Shenzhen, China +1
Structured-Condensed Prompt Tuning in Vision-Language Models for Fine-grained Image Recognition
Jul 7, 2026Xinda Liu, Qinyu Zhang, Weiqing Min +2VLM Adaptation Fine-Grained Image Classification
School of Information Science and Technology, Northwest University, Xi’an, Shaanxi, China · Laboratory of Intelligent Information Processing, Institute of Computing Technology, Beijing, China

Jul 21, 2026, cs.LGJ/K move · Enter open · S save

One Model, Many Graphs: Learning over Attributed Graphs across Heterogeneous Modalities with Vision-Language Models

Jiayi Yang, Yifang Chen, Yuanfu Sun, Jiajin Liu+1

New York University Shanghai · New York University

Graph Representation Learning VLM Adaptation Text-Attributed Graph Learning Multimodal Graph Learning

Open paper AskPDF arXiv

Vision-language models (VLMs) provide a unified representation space for textual and visual information, yet their potential as general-purpose backbones for graph-structured data remains largely unexplored. In practice, attributed graphs exhibit substantial modality heterogeneity: some graphs contain only textual node attributes, others only visual attributes, while still others provide both. Existing graph learning approaches are typically designed for fixed modality schemas, requiring separate models for different settings and limiting scalability and cross-graph generalization. To bridge this gap, we present OMG-VLM (One Model, Many Graphs with Vision-Language Models), a unified framework for learning over attributed graphs across heterogeneous modality schemas. OMG-VLM leverages a pretrained VLM as a shared backbone and introduces structure-aware graph adapters that integrate neighborhood information while remaining compatible with the VLM's native embedding space. This design enables effective learning over text-attributed, image-attributed, and multimodal-attributed graphs within a single model. Extensive experiments across diverse domains show that OMG-VLM consistently outperforms state-of-the-art GNN- and LLM-based baselines on attributed graph learning tasks such as node classification and link prediction, while exhibiting strong generalization to unseen graphs and varying modality schemas. The source code is available at https://github.com/Jo-eyang/OMG-VLM.

Momentum

Related topics

Latest papers 395