Vision-Language Alignment

Follow this topic Ask about this topic

Vision-language alignment studies how models connect visual information with words, phrases, and longer descriptions, enabling tasks such as image-text retrieval, visual grounding, segmentation, and captioning. Common approaches include contrastive learning, prompt and adapter tuning, optimal transport, and fine-grained alignment between text tokens and image regions or video frames. Work also applies these methods to medical images, remote sensing, robotics, and multilingual settings, while developing benchmarks and diagnostics for compositional understanding and cross-modal correspondence. A central tension is that stronger semantic alignment can improve matching but may blur local details, amplify language bias, or weaken generalization across domains and tasks.

Latest papers

361 papers33 papers in the last four weeks, up 22% on the four weeks before. 0.2% of all new papers.

LAST: Bridging Vision-Language and Action Manifolds via Gromov-Wasserstein Alignment
May 27, 2026Huaihai Lyu, Chaofan Chen, Yuheng Ji +4Cross-Modal Learning Vision-Language-Action Models
1MAIS, Institute of Automation, Chinese Academy of Sciences. · 2Beijing Academy of Artificial Intelligence. · 3Peking University..
Rethinking Video-Language Model from the Language Input Perspective
May 27, 2026Xiang Fang, Wanlong Fang, Changshuo Wang +2Prompt Sensitivity VLM Adaptation
School of Software Engineering, Huazhong University of Science and Technology · Nanyang Technological University, Singapore · University College London +2
Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning
May 27, 2026Yuting Ma, Lechao Cheng, Xiaohua XuVLM Adaptation Vision-Language Alignment
School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, Anhui, China.
FAST-GOAL: Fast and Efficient Global-local Object Alignment Learning
May 26, 2026Hyungyu Choi, Young Kyun Jang, Chanho EomVLM Adaptation Text-Image Alignment
Department of Virtual Convergence, Graduate School of Advanced Imaging Science, Multimedia & Films (GSAIM), Chung-Ang University, Seoul, South Korea
Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models
May 25, 2026Xiao Liu, Jiaxiang Liu, Boci Peng +6VLM Robustness Prompt Learning
University of Macau · Guangdong Institute of Intelligence Science and Technology · Peking University +4
Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation
May 24, 2026Yangneng Chen, Jing LiVision-Language Models VLM Adaptation
1Harbin Institute of Technology, Shenzhen, China.
Interpretability Transfer from Language to Vision via Sparse Autoencoders
May 24, 2026Alexey Kravets, Da Li, Chuan Li +2VLM Interpretability Sparse Autoencoders
University of Bath, UK · Samsung AI Centre Cam- bridge · Lambda, Inc..
Spatio-Temporal Similarity Volume Aggregation for Open-Vocabulary Action Recognition
May 22, 2026Yerim So, Jiyeong Kim, Jiwon Yoon +1Video Action Recognition Vision-Language Alignment
Ewha Womans University, South Korea
SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentation
May 21, 2026Zhenyu Lu, Liupeng Li, Jinpeng Wang +4Image Segmentation Text-Guided Image Segmentation
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +3
Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling
May 21, 2026David Méndez, Roberto Confalonieri, Natalia Díaz RodríguezCross-Modal Learning VLM Adaptation
Department of Computer Science and Artificial Intelligence, DaSCI Institute, University of Granada, Granada, Spain · Department of Mathematics “Tullio Levi-Civita”, University of Padova, Padova, Italy.
What Semantics Survive the Connector? Diagnosing VLM-to-DiT Alignment in Video Editing
May 20, 2026Hangyu Lin, Chao Wen, Chengming Xu +4Video Diffusion Models Video Editing
HKUST · FDU · ZJU +1
Pareto-Enhanced Portrait Generation: Vision-Aligned Text Supervision for Alignment, Realism, and Aesthetics
May 20, 2026Yunlong Wang, Jinjin Shi, Wenbin Gao +3Diffusion Model Alignment Diffusion Transformer
Xiaomi Corporation Nanjing, China · Xiaomi Corporation Beijing, China
Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
May 18, 2026Xinpeng Dong, Min Zhang, Kairong Han +3Object Hallucination in VLMs Multimodal Large Language Models
Zhejiang University · East China Normal University · Zhejiang University of Science and Technology
See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding
May 18, 2026Boyuan Sun, Bowen Yin, Yuanming Li +2Cross-Modal Learning Video Object Segmentation
VCIP, CS, Nankai University · Tongyi Lab, Alibaba Group · NKIARI, Shenzhen Futian
GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representations
May 18, 2026Zesheng Li, Chengchang Pan, Honggang QiVision-Language Models Representation Learning
University of the Chinese Academy of Sciences, China
SHED: Style-Homogenized Embedding Alignment for Domain Generalization
May 16, 2026Kai Gan, Tong WeiVLM Robustness VLM Adaptation
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China
GeoWorld-VLM: Geometry from World Models for Vision-Language Models
May 15, 2026Renjie Gu, Kaichen Zhou, Yan Luo +1Visual Spatial Reasoning VLM Adaptation
Harvard AI and Robotics Lab · Kempner Institute for the Study of Natural and Artificial Intelligence · Harvard University
Res $^2$ CLIP: Few-Shot Generalist Anomaly Detection with Residual-to-Residual Alignment
May 15, 2026Xinyue Liu, Jianyuan Wang, Biao Leng +1Few-Shot Anomaly Detection VLM Adaptation
Beihang University · University of Science and Technology Beijing · Beijing Jiaotong University
Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation
May 15, 2026Chenhao Wang, Yingrui Ji, Yu Meng +1Image Segmentation Open-Vocabulary Semantic Segmentation
Aerospace Information Research Institute, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Zhejiang University
BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
May 15, 2026Huanyang Tong, Kai Liu, Fangjun Kuang +1VLM Robustness Prompt Learning
Wenzhou University · Wenzhou Business College
Deep Pre-Alignment for VLMs
May 14, 2026Tianyu Yu, Kechen Fang, Zihao Wan +5Vision-Language Models Large Vision-Language Models
1Tsinghua University · 2Taobao & Tmall Group of Alibaba · 3Shanghai Qi Zhi Institute.
Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning
May 13, 2026Hao Sun, Zi-Jun Ding, Da-Wei ZhouClass-Incremental Learning VLM Adaptation
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University
Revealing the Gap in Human and VLM Scene Description through Counterfactual Semantic Saliency
May 13, 2026Ziqi Wen, Parsa Madinei, Miguel P. EcksteinVLM Evaluation Vision-Language Models
Department of Computer Science, University of California, Santa Barbara · Department of Psychological and Brain Sciences, University of California, Santa Barbara
VIP: Visual-guided Prompt Evolution for Efficient Dense Vision-Language Inference
May 12, 2026Hao Zhu, Shuo Jin, Wenbin Liao +4Image Segmentation Vision-Language Models
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · XJTLU +1
Multimodal Abstractive Summarization of Instructional Videos with Vision-Language Models
May 12, 2026Maham Nazir, Muhammad Aqeel, Richong Zhang +1Video Summarization Video-Language Models
Beihang University, Beijing, China · University of Verona, Italy
Emergent Communication between Heterogeneous Visual Agents through Decentralized Learning
May 12, 2026Mikako Ochiai, Masatoshi Nagano, Tadahiro TaniguchiMulti-Agent Systems Vision-Language Alignment
Graduate School of Informatics, Kyoto University, Kyoto 606-8501, Japan
Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning
May 12, 2026Yaze Zhao, Yicong Liu, Yixiong Zou +2Source-Free Cross-Domain Few-Shot Learning Source-Free Domain Adaptation
VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Models
May 11, 2026Hao Wang, Xiaobao Wei, Jingyang He +103D Spatial Reasoning Spatial Grounding for VLA Models
1Peking University, Beijing, China · 2Beijing Innovation Center of Humanoid Robotics, Beijing, China
Thermal-Det: Language-Guided Cross-Modal Distillation for Open-Vocabulary Thermal Object Detection
May 11, 2026Yasiru Ranasinghe, Elim Schenck, Florence Yellin +3Open-Vocabulary Object Detection Cross-Modal Knowledge Distillation
Johns Hopkins University · Kitware · DEVCOM Army Research Laboratory
DAPE: Dynamic Non-uniform Alignment and Progressive Detail Enhancement Techniques for Improving the Performance of Efficient Visual Language Models
May 9, 2026Mengyuan Tian, Qiyan Zhao, Yanan Wang +1VLM Adaptation Efficient VLM Inference
The Wang Yanan Institute for Studies in Economics, Xiamen University, Xiamen 361005, China · Fujian Key Laboratory of Pattern Recognition and Image Understanding, School of Computer and Information Engineering, Xiamen University of Technology, Xiamen 361024, China

May 27, 2026J/K move · Enter open · S save

LAST: Bridging Vision-Language and Action Manifolds via Gromov-Wasserstein Alignment

Huaihai Lyu, Chaofan Chen, Yuheng Ji, Xiansheng Chen+3

1MAIS, Institute of Automation, Chinese Academy of Sciences. · 2Beijing Academy of Artificial Intelligence. · 3Peking University..

Cross-Modal Learning Vision-Language-Action Models Vision-Language Alignment

Open paper AskPDF arXiv

We take a Gromov-Wasserstein perspective on Vision-Language-Action (VLA) learning, where the goal is to make the relational geometry of action representations compatible with the semantic geometry of VL embeddings. However, this alignment is non-trivial due to the mathematical heterogeneity between the domains: the semantic space of vision-language is topologically linear and isotropic, whereas the physical manifold of robotic action is non-Euclidean and anisotropic. Their disjoint metric structures render direct regression ill-posed. To resolve this incompatibility, we introduce LAST (Lie-algebraic Action Space Tokenizer), which reconstructs the action space to establish local metric compatibility with the VL modality via a two-stage transformation: (1) Global Topological Linearization: linearizing the action manifold via Lie-algebraic mapping, converting trajectories into a fixed-length, physically additive representation. (2) Local Metric Discretization: hierarchically discretizing the representation into schemas and whitened residuals, yielding approximately isotropic local charts that are statistically aligned with the semantic metric. By resolving the structural mismatch at both global and local levels, LAST enables VLA models with superior convergence and generalizability.