Vision-Language Alignment

Follow this topic Ask about this topic

Vision-language alignment studies how models connect visual information with words, phrases, and longer descriptions, enabling tasks such as image-text retrieval, visual grounding, segmentation, and captioning. Common approaches include contrastive learning, prompt and adapter tuning, optimal transport, and fine-grained alignment between text tokens and image regions or video frames. Work also applies these methods to medical images, remote sensing, robotics, and multilingual settings, while developing benchmarks and diagnostics for compositional understanding and cross-modal correspondence. A central tension is that stronger semantic alignment can improve matching but may blur local details, amplify language bias, or weaken generalization across domains and tasks.

Latest papers

361 papers33 papers in the last four weeks, up 22% on the four weeks before. 0.2% of all new papers.

Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation
May 9, 2026Hoang M. Truong, Hai Nguyen-Truong, Dang HuynhVLM Adaptation Hierarchical Representation Learning
Fulbright University Vietnam
Zero-Shot Chinese Character Recognition via Global-Local Dual-Branch Alignment and Hierarchical Inference
May 9, 2026Wei Cao, Hao Xu, Xiaolei DiaoZero-Shot Learning Efficient VLM Inference
1Jilin University, China · University College London, UK
CapCLIP: A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysis
May 8, 2026Haroon Wahab, Irfan Mehmood, Hassan UgailEndoscopy Medical Image Analysis
School of Computer Science, AI and Electronics Faculty of Engineering and Digital Technologies University of Bradford Bradford, UK · School of Management Faculty of Mgmt, Law & Social Sciences University of Bradford Bradford, UK · Centre for Visual Computing and Intelligent Systems University of Bradford Bradford, UK
Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment
May 8, 2026Jerry Jiang, Haowen Sun, Denis Gudovskiy +43D Spatial Reasoning 3D Representation Learning
Tsinghua University · Panasonic AI Lab · Panasonic DX-CPS +1
Weather-Robust Scene Semantics with Vision-Aligned 4D Radar
May 8, 2026Kali Hamilton, Christoffer HeckmanCross-Modal Learning VLM Robustness
Autonomous Robotics and Perception Group at the University of Colorado Boulder, Boulder, CO 80309, USA.
Learning to Track Instance from Single Nature Language Description
May 8, 2026Yaozong Zheng, Bineng Zhong, Qihua Liang +3Visual Object Tracking Vision-Language Alignment
Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education Guangxi Normal University, Guilin 541004, China · University Engineering Research Center of Educational Intelligent Technology Guangxi Normal University, Guilin 541004, China · University of Southampton, Southampton, SO17 1BJ, United Kingdom
DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistency
May 7, 2026Shuyang Jiang, Nan Yu, Yiming Zhang +2Contrastive Learning Vision-Language Alignment
University of California, Los Angeles · Aimaikj · HFIPS, Chinese Academy of Sciences +2
T2I-VeRW: Part-level Fine-grained Perception for Text-to-Image Vehicle Retrieval
May 7, 2026Xiao Wang, Ziwen Wang, Weizhe Kong +5Fine-Grained Image Retrieval Cross-Modal Retrieval
School of Computer Science and Technology, Anhui University, Hefei 230601, China · School of Artificial Intelligence, Anhui University, Hefei 230601, China
When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
May 7, 2026Harshvardhan Saini, Samyak Jha, Yiming Tang +1VLM Hallucination Mitigation Vision-Language Models
Indian Institute of Technology Dhanbad · National University of Singapore
Prompt-Anchored Vision-Text Distillation for Lifelong Person Re-identification
May 6, 2026Wen Wen, Hao Chen, Shiliang ZhangCross-Modal Knowledge Distillation Continual Learning
University of Electronic Science and Technology of China, China · Harbin Institute of Technology, Shenzhen, China · Peking University, China
HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
May 6, 2026Esra Dönmez, Pascal Tilli, Hsiu-Yu Yang +2Cross-Modal Learning Image Matching
Institute for Natural Language Processing, University of Stuttgart
Reward-Guided Semantic Evolution for Test-time Adaptive Object Detection
May 6, 2026Lihua Zhou, Mao Ye, Xiatian Zhu +7Open-Vocabulary Object Detection Test-Time Adaptation
Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China · School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China · Surrey Institute for People-Centred Artificial Intelligence, CVSSP, University of Surrey, Guildford, UK +3
The Detector Teaches Itself: Lightweight Self-Supervised Adaptation for Open-Vocabulary Object Detection
May 5, 2026Yazhe Wan, Changjae OhOpen-Vocabulary Object Detection VLM Adaptation
Queen Mary University of London · Queen Mary University of London, London, UK
Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models
May 5, 2026Shule Lu, Yujing Wang, Hainan Zhang +5Reward Modeling Preference Optimization
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing · Institute of Artificial Intelligence, Beihang University, China · MAIS, Institute of Automation, Chinese Academy of Sciences, China +2
MASRA: MLLM-Assisted Semantic-Relational Consistent Alignment for Video Temporal Grounding
May 5, 2026Ran Ran, Jiwei Wei, Shuchang Zhou +5Temporal Video Grounding Cross-Modal Alignment
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Cyberspace Security, Hainan University, Haikou, China
HIMMEL: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding
May 4, 2026Haopeng Jin, Wenlong Zhao, Tao Yu +8Video Understanding Video-Language Models
Tencent · University of Chinese Academy of Sciences · Beijing Forestry University
LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment
May 4, 2026Junyi Hu, Qiji Zhou, Lei Zhang +1VLM Robustness Efficient VLM Inference
Beijing Jiaotong University · Westlake University · Rochester Institute of Technology
Research on Vision-Language Question Answering Models for Industrial Robots
May 2, 2026Ping Li, Bartlomiej BrzozkaVisual Question Answering Robotics
Shandong Management University, Jinan, Shandong Province, 250357, China · Maria Curie-Sklodowska University, Faculty of Mathematics, Physics and Computer Science, 20-031 Lublin, Poland
Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance
May 2, 2026Muyang Li, Yucheng Liu, Jianbo Ma +3Vision-Language Models Vision-Language Alignment
1Sydney AI Centre, The University of Sydney · 2Dolby Laboratories · 3TMLR Group, Hong Kong Baptist University
Static and Dynamic Graph Alignment Network for Temporal Video Grounding
May 1, 2026Zhanjie Hu, Bolin Zhang, Jianhua Wang +5Temporal Video Grounding Temporal GNNs
Faculty of Electronic Engineering and Computer Science, Ningbo University, Ningbo, China · College of Computer Science, Inner Mongolia University, Hohhot, China · College of Computing and Engineering, Hunan Normal University, Changsha, China +4
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3
RIHA: Report-Image Hierarchical Alignment for Radiology Report Generation
Apr 30, 2026Yucheng Chen, Yang Yu, Yufei Shi +3Radiology Report Generation Radiology VLMs
MedVisAI Lab, Lee Kong Chian School of Medicine, Nanyang Technological University (NTU), and Centre of AI in Medicine, Singapore · Department of Computer Science and Engineering, the Chinese University of Hong Kong, Hong Kong SAR, China · Department of Machine Intellection, Institute for Infocomm Research (I2R), Agency for Science, Technology and Research, (A*STAR), Singapore
COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts
Apr 30, 2026Bingli Wang, Huanze Tang, Haijun Lv +5Vision-Language Models Vision-Language Grounding
1Southeast University · 2Shanghai AI Laboratory
Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning
Apr 29, 2026Junwon You, Mihyun Jang, Sangwoo Mo +1Cross-Modal Learning Cross-Modal Representation Learning
KAIST · POSTECH
SycoPhantasy: Quantifying Sycophancy and Hallucination in Small Open Weight VLMs for Vision-Language Scoring of Fantasy Characters
Apr 27, 2026Arya Shah, Deepali Mishra, Chaklam SilpasuwanchaiVLM Evaluation Vision-Language Models
IIT Gandhinagar Gandhinagar, India · IIT Kanpur Kanpur, India · Asian Institute of Technology Bangkok, Thailand
Selective Contrastive Learning For Gloss Free Sign Language Translation
Apr 24, 2026Changhao Lai, Rui Zhao, Xuewen Zhong +2Contrastive Learning Sign Language Translation
School of Informatics, Xiamen University, China · 2Key Lab of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian-Taiwan (XMU), Ministry of Culture and Tourism, China · 3National Language Resources Monitoring and Research Center for Education and Teaching Media, Xiamen University, China
Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion
Apr 23, 2026Eghbal A. Hosseini, Brian Cheung, Evelina Fedorenko +1Cross-Modal Alignment Vision-Language Alignment
Department of Brain and Cognitive Sciences, MIT · CSAIL, MIT · Center for Neural Science, NYU +1
Latent Denoising Improves Visual Alignment in Large Multimodal Models
Apr 23, 2026Dhruv Parikh, Jacob Fein-Ashley, Rajgopal Kannan +1VLM Robustness Multimodal Large Language Models
University of Southern California Los Angeles, CA, USA · DEVCOM ARL Army Research Office USA
ReCAPA: Hierarchical Predictive Correction to Mitigate Cascading Failures
Apr 23, 2026Xiyin Zeng, Yuyu Sun, Haoyang Li +2Vision-Language-Action Models Vision-Language Alignment
1Hong Kong University of Science and Technology (Guangzhou) · 2South China Normal University
EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training
Apr 21, 2026Yiyang Du, Zhanqiu Guo, Haochen Zhang +5VLM Adaptation Vision-Language-Action Models
Language Technologies Institute, Carnegie Mellon University · Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI)

May 9, 2026J/K move · Enter open · S save

Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation

Hoang M. Truong, Hai Nguyen-Truong, Dang Huynh

Fulbright University Vietnam

VLM Adaptation Hierarchical Representation Learning Open-Vocabulary Semantic Segmentation Vision-Language Alignment Hyperbolic Representation Learning

Open paper AskPDF arXiv

Open-vocabulary semantic segmentation requires adapting image-level vision-language models such as CLIP to dense pixel-level prediction, which is challenging due to the mismatch between hierarchical structure and semantic alignment in the embedding space. While recent works leverage hyperbolic geometry to model hierarchical relationships, they align embeddings across hierarchical levels but overlook semantic misalignment among embeddings within the same level. In this work, we propose HyRo, a hyperbolic fine-tuning framework that decouples hierarchical and semantic alignment in the Poincaré ball model. HyRo aligns hierarchical levels by adjusting the hyperbolic radius and refines semantic relationships through angular alignment using an orthogonal transformation that theoretically preserves the hyperbolic radius. Experiments on standard open-vocabulary semantic segmentation benchmarks demonstrate that HyRo achieves state-of-the-art performance over prior methods.