Vision-Language Alignment

Follow this topic Ask about this topic

Vision-language alignment studies how models connect visual information with words, phrases, and longer descriptions, enabling tasks such as image-text retrieval, visual grounding, segmentation, and captioning. Common approaches include contrastive learning, prompt and adapter tuning, optimal transport, and fine-grained alignment between text tokens and image regions or video frames. Work also applies these methods to medical images, remote sensing, robotics, and multilingual settings, while developing benchmarks and diagnostics for compositional understanding and cross-modal correspondence. A central tension is that stronger semantic alignment can improve matching but may blur local details, amplify language bias, or weaken generalization across domains and tasks.

Latest papers

361 papers33 papers in the last four weeks, up 22% on the four weeks before. 0.2% of all new papers.

EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment
Aug 5, 2026Zhenyu Yi, Jianwei Xu, Yue Hu +6Medical Imaging Foundation Models Medical VLMs
1DAMO Academy, Alibaba Group · 3Shanghai Jiao Tong University · 4Hupan Lab +3
Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentation
Aug 4, 2026Wanli Ma, Jiangwen Lu, Qinmu Peng +1Open-Vocabulary Semantic Segmentation Vision-Language Alignment
School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · National Anti-counterfeit Engineering Research Center, Huazhong University of Science and Technology, Wuhan 430074, China · School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China
Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach
Aug 4, 2026Tianbao Jiang, Weicong Ni, Gerard de Melo +1Large Vision-Language Models VLM Reasoning
East China Normal University Shanghai, China · Hasso Plattner Institute Potsdam, Germany · University of Potsdam Potsdam, Germany +1
Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression
Aug 3, 2026Tianyu Liang, Xiangxi Zheng, Yilin Wang +1VLM Adaptation Multimodal Token Compression
Southeast University Nanjing, China · Nanjing University Nanjing, China · Zhejiang University Hangzhou, China +1
CAVE: Competence-Aware Visual Boundary Evidence Alignment for Video Temporal Grounding
Aug 3, 2026Wei Jia, Zhicong Lu, Yu Chen +5Temporal Video Grounding Vision-Language Alignment
AMAP, Alibaba Group · University of Chinese Academy of Sciences
Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning
Aug 3, 2026Qianlong Yang, Bowen Ye, Xianda Guo +4VLM Robustness Multimodal Large Language Models
China University of Petroleum (East China) Qingdao, Shandong, China · Great Wall Motor · Shanghai Jiao Tong University Shanghai, China +3
PixVL: Self-Supervised Training of Pixel-Level MLLMs via a Unified Mask--Text Consistency Cycle
Aug 2, 2026Yicheng Xiao, Haoxuan Ma, Caorui Li +7Self-Supervised Learning VLM Adaptation
Institute of Automation, Chinese Academy of Sciences · Nanjing University · Southeast University +3
Entity-Faithful Repair of Synthetic Supervision for Zero-Shot Image Captioning
Aug 2, 2026Zhiyue Liu, Wenkai Zhou, Jian Qin +1Synthetic Data Augmentation Image Captioning
School of Computer, Electronics and Information Guangxi Key Laboratory of Multimedia Communications and Network Technology Guangxi University Nanning, Guangxi, China
Through the LENS: Local Geometric Decomposition of Vision-Language Model Representations
Aug 1, 2026Shalom Kachko, Raz Lapid, Margarita Vald +2Vision-Language Models Representation Geometry in VLMs
1Ben-Gurion University of the Negev · 2Intuit Inc. · 3Deepkeep
Learning to See Locally and Align Clinically with Pathology Semantics for Radiology Report Generation
Jul 31, 2026Xuan Cuong NgoRadiology Report Generation VLM Robustness
QR-Structured Thermal Triggers for Targeted Semantic Attacks on Infrared Vision-Language Models
Jul 31, 2026Xiang Chen, Yingying Zhao, Chao Li +7VLM Robustness Adversarial Attacks on VLMs
Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning
Jul 31, 2026Duy Tran Thanh, Thien-Phuc Doan, Long Nguyen-Vu +1Image Captioning Vision-Language Alignment
AI Platform OneNexus, OneMount · School of Electronic Engineering, Soongsil University, Seoul 06978, South Korea · AI Laboratory, MoAdata, Seoul, South Korea +1
DualAnchor: Preserving Language Priors and Improving Lexical Fidelity in Gloss-Free Sign Language Translation
Jul 30, 2026Hongbin Zhang, Junhao Liu, Xuefeng Bai +3Sign Language Translation Vision-Language Alignment
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China
Anatomy Contextualized Adaption of CT Foundation Models
Jul 29, 2026Roshan Kenia, Stephanie L McNamara, William LotterMedical Imaging Foundation Models VLM Adaptation
Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Data Science, Dana-Farber Cancer Institute, Boston, MA · Department of Radiology, Massachusetts General Hospital, Boston, MA +2
SciFigAlign: Scoring Scientific Figures by Fine-tuned Alignment of Visuals with Manuscript Evidence
Jul 29, 2026Chuanzhi Xu, Zihan Deng, Huiqi Liang +4Scholarly Peer Review Image Quality Assessment
The University of Sydney · The University of Hong Kong · Indiana University +2
SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representation
Jul 29, 2026Yunzhan Fu, Enyu Bao, Xiangyu Shen +4Radiology VLMs Medical VLMs
School of Computer Science, Hangzhou Dianzi University, China
SAM3D-Guided Object-Centric Representation Alignment for Vision-Language-Action Models
Jul 28, 2026Zonghe Liu, Shanyuan Jie, Xiaoquan Sun +4Long-Horizon Robotic Manipulation Object-Centric Representation Learning
University of Hong Kong · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Huazhong University of Science and Technology +2
Fine-Grained Food Image Understanding via Target-Aware Data Alignment
Jul 28, 2026Jui-Feng Chi, Wei-Lun Chu, Bruce Coburn +2Fine-Grained Image Retrieval Fine-Grained Image Classification
Elmore Family School of Electrical and Computer Engineering Purdue University, West Lafayette, IN, USA
TraceCLIP: Recovering Local Semantics from Patch-to-CLS Contributions
Jul 28, 2026Xinran Liu, Shouqian Shi, Yutong Chen +3Open-Vocabulary Semantic Segmentation Vision-Language Alignment
What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features
Jul 25, 2026Chen-Yi Lu, Yueh-Shao Chen, Somali ChaterjiVision-Language Models VLM Adaptation
Purdue University, West Lafayette, IN 47907, USA
EVL-MCoT: Enhanced Vision-Language Multi-CoT for Harmful Meme Detection
Jul 24, 2026Hao Yang, Jin Wang, Xuejie ZhangContent Moderation VLM Reasoning
School of Information Science and Engineering, Yunnan University, Kunming, China
TextSLIP: Text Self-Supervised CLIP for Medical Report Generation
Jul 24, 2026Haoyu Jiang, Ziping CongRadiology Report Generation Self-Supervised Learning
Centre for Artificial Intelligence and Robotics (CAIR) · Hong Kong Institute of Science & Innovation · Chinese Academy of Sciences
DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation
Jul 23, 2026Sung-Hoon Yoon, Hoyong Kwon, Changgyoon Oh +1VLM Adaptation Open-Vocabulary Semantic Segmentation
Multimodal Intelligence and Perception Lab., DGIST, Republic of Korea · Visual Intelligence Lab., KAIST, Republic of Korea
Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrieval
Jul 21, 2026Jihyun Lee, Cheol-Ho Cho, Woojin Jun +2Temporal Video Grounding Vision-Language Alignment
Sungkyunkwan University, Suwon, Republic of Korea
KALE: Kernel Alignment with Loss Equilibration for Stable CLIP-DINOv2 Alignment at Web Scale
Jul 21, 2026Michał PawłowiczVLM Adaptation Adaptive Loss Weighting
When Physical Preferences Meet Semantic Constraints: Physical and Semantic Direct Preference Optimization for Text-to-Video Generation
Jul 18, 2026Siwei Meng, Yawei Luo, Shu Zhang +1Physical Consistency in Video Generation Direct Preference Optimization
Computer Science and Engineering · University of Nevada, Reno · Reno, NV, USA +5
SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning
Jul 15, 2026Cheng Tang, Junzhi Ning, Min Cen +9Reinforcement Learning with Verifiable Rewards Visually Grounded Reasoning
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Sichuan University +2
AspectCLIP: Optimizing CLIP Representation Space via Aspect-Guided Consistency Regularization
Jul 15, 2026Yiyang Yao, Shanglin Liu, Jianming Lv +4Vision-Language Alignment Multimodal Contrastive Learning
School of Computer Science and Technology, South China University of Technology, Guangzhou, China
Fine-grained CLIP fine-tuning with self-annotated region alignment
Jul 15, 2026Chenyang Zhao, Wei Lin, Antoni B. Chan +1VLM Adaptation Vision-Language Grounding
Department of Computer Science, City University of Hong Kong · Division of Social Science, Hong Kong University of Science & Technology
Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment
Jul 15, 2026Dwip Dalal, Shivansh Patel, Chahit Jain +8VLA Model Training Vision-Language-Action Models
University of Illinois Urbana-Champaign · Texas A&M University · University of California, Irvine

Aug 5, 2026J/K move · Enter open · S save

EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment

Zhenyu Yi, Jianwei Xu, Yue Hu, Zhongwei Qiu+5

1DAMO Academy, Alibaba Group · 3Shanghai Jiao Tong University · 4Hupan Lab+3

Medical Imaging Foundation Models Medical VLMs Vision-Language Alignment Medical Image Representation Learning

Open paper AskPDF arXiv

The development of foundation models (FMs) is crucial for advancing endoscopic image analysis. However, existing endoscopy FMs mainly rely on self-supervised learning from uni-modal images or videos, overlooking the rich semantic knowledge contained in clinical reports. Furthermore, effectively leveraging these records is hindered by a fundamental modality gap: structured anatomical descriptions are not naturally mapped to specific frames within the high-redundancy, uncurated visual streams. In this paper, we present EndoVLM, a novel vision-language FM pre-trained on over 348K endoscopic examinations, each pairing a clinical report with its corresponding image collection. An Anatomy-Guided Sparse Pooling mechanism utilizes textual descriptions as queries to drive sparse attention, efficiently aggregating semantically salient frames into anatomy-specific visual representations across redundant image-sets. Next, a Progressive Semantic-Aware Alignment strategy models clinical taxonomy (anatomy and pathological status) via structured soft targets, bridging the gap from global patient-level matching to fine-grained localized alignment. Finally, a Semantic-Concentrated Masked Autoencoder is applied exclusively to these semantic-rich frames, integrating low-level visual precision with robust high-level semantic representation. Extensive experiments across various downstream tasks demonstrate that EndoVLM outperforms existing foundation models and remains competitive with task-specific methods. Remarkably, EndoVLM also exhibits robust zero-shot generalization capabilities, highlighting its potential for broader clinical application.