Vision-Language Alignment

Follow this topic Ask about this topic

Vision-language alignment studies how models connect visual information with words, phrases, and longer descriptions, enabling tasks such as image-text retrieval, visual grounding, segmentation, and captioning. Common approaches include contrastive learning, prompt and adapter tuning, optimal transport, and fine-grained alignment between text tokens and image regions or video frames. Work also applies these methods to medical images, remote sensing, robotics, and multilingual settings, while developing benchmarks and diagnostics for compositional understanding and cross-modal correspondence. A central tension is that stronger semantic alignment can improve matching but may blur local details, amplify language bias, or weaken generalization across domains and tasks.

Latest papers

361 papers33 papers in the last four weeks, up 22% on the four weeks before. 0.2% of all new papers.

FusionRS: A Large-Scale RGB-Infrared Remote Sensing Dataset for Dual-Modal Vision-Language Foundation Models
Jun 15, 2026Jiaju Han, Ben Zhang, Xuemeng Sun +6Remote Sensing Image Understanding Remote Sensing Image Captioning
China University of Petroleum-Beijing at Karamay · University of Electronic Science and Technology of China · Tianjin University
Decoupling Semantics from Distortions: Multi-Scale Two-Stream Vision-Language Alignment for AI-Generated Image Quality Assessment
Jun 15, 2026Zijie MengVLM Evaluation Image Quality Assessment
Peking University
Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality
Jun 11, 2026Wei Li, Zhen Huang, Xinmei TianCross-Modal Learning Vision-Language Models
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · Independent Researcher
Language-Guided Abstraction for Visual Reasoning
Jun 11, 2026Xu-Jing Ye, Yuan-Gen Wang, Ruping WangVisual Reasoning Vision-Language Alignment
School of Artificial Intelligence, Guangzhou University, Guangzhou, 510006, China · Traditional Chinese Medicine Hospital of Zengcheng District, Guangzhou, 511399, China
ECA: Efficient Continual Alignment for Open-Ended Image-to-Text Generation
Jun 10, 2026Jiangtao Kong, Peijun Zhao, Chun-Fu Chen +4Large Vision-Language Models Incremental Learning
Department of Computer Science, William & Mary, Virginia, USA · Global Technology Applied Research, JPMorgan-Chase, USA · Mohamed bin Zayed University of Artificial Intelligence, Masdar City, Abu Dhabi
Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs
Jun 10, 2026Shayan Mohammadizadehsamakosh, Pritam Sarkar, Leonid Sigal +2Medical VLMs Medical Image Grounding
York University · Vector Institute · University of British Columbia +1
VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio
Jun 10, 2026Hao Zhang, Qinran Lin, Linqi Song +1Open-Vocabulary Object Detection Detection Transformer
Chongqing University, Chongqing, China · City University of Hong Kong, Hong Kong, China
Improving Text-Instance Alignment Of Foreground Conditioned Out-Painting Via Customized Concept Embedding
Jun 9, 2026Yihao Zhao, Xuan Han, Bin He +1Conditional Image Generation Vision-Language Alignment
College of Electronic and Information Engineering, Tongji University, Shanghai, China · State Key Laboratory of Autonomous Intelligent Unmanned Systems, Frontiers Science Center for Intelligent Autonomous Systems, Ministry of Education, Shanghai, China
TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment
Jun 5, 2026Sweta Mahajan, Sukrut Rao, Jiahao Xie +2Vision-Language Retrieval Vision-Language Alignment
Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany · Department of Language Science and Technology, Saarland University, Saarbrücken, Germany
MotionEnhancer: Leveraging Video Diffusion for Motion-Enhanced Vision-Language Models
Jun 5, 2026Yifan Xu, Chao Zhang, Ruifei Ma +4VLM Adaptation Vision-Language Alignment
School of Computer Science and Engineering, Beihang University · 2Beijing Digital Native Digital City Research Center · School of Computer Science, Peking University +1
Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain
Jun 3, 2026Alessandro Gambetti, Qiwei Han, Cláudia Soares +1VLM Evaluation Modality Imbalance
NOVA School of Science and Technology · Lisbon, Portugal · Nova School of Business and Economics +2
Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models
Jun 3, 2026Shuwen Yu, Zhanxuan Hu, Yi Zhao +2Cross-Modal Representation Learning Vision Foundation Model Adaptation
Yunnan Normal University, Kunming, China · Kunming University of Science and Technology, Kunming, China
KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models
Jun 2, 2026Youqi Wu, Mohammad Jalali, Farzan FarniaKernel Methods Multimodal Foundation Models
Department of Computer Science and Engineering, The Chinese University of Hong Kong.
Visual Instruction Tuning Aligns Modalities through Abstraction
Jun 2, 2026Luis Palacios, Lorenzo Basile, Diego Doimo +1VLM Adaptation VLM Interpretability
Area Science Park, Trieste, Italy
GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations
Jun 2, 2026Jonggwon Park, Seongeun Lee, Junhyun Park +6Radiology VLMs Medical Image Segmentation
DEEPNOID Inc., Seoul, South Korea · Department of Artificial Intelligence and Data Science, Sejong University, Seoul, South Korea
Towards Precise Intent-Aligned VLA Aerial Navigation via Expert-Guided GRPO
Jun 1, 2026Tianyang Chen, Wenjun Li, Xin zhou +2UAV Navigation Group Relative Policy Optimization
1Zhejiang University · 2Differential Robotics
Cross-modal linkage risk in clinical vision-language models
Jun 1, 2026Soroosh Tayebi Arasteh, Mahshad Lotfinia, Sven Nebelung +1Medical VLMs Privacy Protection in VLMs
Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany · Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
ROGLE: Robust Global-Local Alignment with Automated Region Supervision for Text-Based Person Search
Jun 1, 2026Chaodong Jia, Zequn Xie, Xibei Jia +3Text-Based Person Search Image-Text Retrieval
FlowOVD: Learning Generative Latent Flows for Zero-shot Open-vocabulary Detection
May 30, 2026Yao Wei, Andrea Cavallaro, Changjae OhOpen-Vocabulary Object Detection Vision-Language Alignment
1Queen Mary University of London · 2EPFL
Detect Before You Leap: Mirage Detection in Vision-Language Models
May 29, 2026Md. Shaown Miah, S. M. Taiabul Haque, Syed Ishtiaque Ahmed +1Hallucination Detection in VLMs Vision-Language Models
Indiana University Indianapolis · University of Toronto
How can embedding models bind concepts?
May 29, 2026Arnas Uselis, Darina Koishigarina, Seong Joon OhMultimodal Embedding Vision-Language Alignment
University of Tübingen, Tübingen · 2KAIST AI
HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model
May 29, 2026Xiang Zhu, Puzhen Yuan, Yichen Liu +1Cross-Embodiment Robot Learning Robotic Manipulation
Institute for Interdisciplinary Information Sciences, Tsinghua University, China · Shanghai Qi Zhi Institute, China
Attend to Evidence: Evidence-Anchored Spatial Attention Supervision for Multimodal RLVR
May 29, 2026Ruina Hu, Chen Wang, Lai Wei +5Reinforcement Learning with Verifiable Rewards Visually Grounded Reasoning
1Harbin Institute of Technology · 2Zhongguancun Academy · 4Nankai University +4
Seeing Before Agreeing: Aligning Multi-Agent Consensus with Visual Evidence
May 29, 2026Yuhan Wang, Shuochen Chang, Yalin Feng +8Multi-Agent Collaboration VLM Reasoning
Peking University · Shanghai Jiao Tong University · Nanyang Technological University +2
LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
May 28, 2026Feng Han, Zhixiong Zhang, Zheming Liang +2Cross-Modal Learning VLM Robustness
Fudan University · Shanghai Innovation Institute · Shanghai Jiao Tong University +1
Unveiling the Visual Counting Bottleneck in Vision-Language Models
May 28, 2026Xingzhou Pang, Yifan Hou, Junling Wang +1Object Counting Latent Visual Reasoning
Department of Computer Science, ETH Zürich
Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning
May 28, 2026Shuai Yi, Yixiong Zou, Yuhua Li +1Source-Free Cross-Domain Few-Shot Learning Source-Free Domain Adaptation
Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.
FlowSeg: Dynamic Semantic Guidance for LLM-Conditioned Segmentation
May 28, 2026Zekang Zhang, Guangyu Gao, Youyun Tang +7Text-Guided Image Segmentation Referring Image Segmentation
School of Computer Science, Beijing Institute of Technology · School of Computer Science, University of Birmingham · WEI Lab, Institute of Information Science, Beijing Jiaotong University +1
GAP3D: Generative Alignment of VLM Latents to Patch-Level Embeddings for 3D Generation
May 27, 2026Polytimi Anna Gkotsi, Andrii Zadaianchuk, Mohammad Mahdi DerakhshaniDiffusion Model Alignment 3D Asset Generation
University of Amsterdam, The Netherlands
LV-OSD: Language-Vision-Complementary Open-Set Object Detection
May 27, 2026Yupeng Zhang, Ruize Han, Wei Feng +2Open-Vocabulary Object Detection Object Detection
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen 518107, Guangdong, China

Jun 15, 2026J/K move · Enter open · S save

FusionRS: A Large-Scale RGB-Infrared Remote Sensing Dataset for Dual-Modal Vision-Language Foundation Models

Jiaju Han, Ben Zhang, Xuemeng Sun, Qike Zhang+5

China University of Petroleum-Beijing at Karamay · University of Electronic Science and Technology of China · Tianjin University

Remote Sensing Image Understanding Remote Sensing Image Captioning Remote Sensing VLMs Multimodal Pretraining Remote Sensing Vision-Language Alignment

Open paper AskPDF arXiv

Remote sensing vision-language models have advanced Earth observation understanding, but most existing work remains centered on RGB imagery, leaving the complementary information in infrared data underexplored. Infrared images provide distinctive cues, including thermal intensity structures, object boundaries, and illumination-invariant scene features, which can enrich visual-language learning beyond conventional RGB observations. However, a large-scale RGB-infrared-text dataset for remote sensing vision-language modeling is still absent. To address this gap, we introduce FusionRS, the first large-scale RGB-infrared-text dataset designed for dual-modal vision-language learning in remote sensing. FusionRS is constructed by translating diverse public RGB remote sensing images into infrared-style counterparts, forming aligned RGB-IR image pairs. Each pair is associated with conventional scene captions and IR-aware captions that explicitly describe infrared-specific visual properties while preserving semantic content. Based on FusionRS, we train dual-modal vision-language foundation models for RGB-IR joint understanding. We first train CLIP-style models for RGB-IR-text alignment, and then fine-tune generative VLMs for dual-modal RGB-IR captioning. Experiments show that FusionRS improves RGB-IR alignment, infrared-to-text retrieval, and dual-modal captioning over RGB-only and non-IR-aware training settings. Ablation studies further verify that IR-aware captions are crucial for strengthening infrared-language alignment, highlighting the importance of modality-specific textual supervision for more scalable RGB-infrared remote sensing vision-language representation learning.