Vision-Language Alignment

Follow this topic Ask about this topic

Vision-language alignment studies how models connect visual information with words, phrases, and longer descriptions, enabling tasks such as image-text retrieval, visual grounding, segmentation, and captioning. Common approaches include contrastive learning, prompt and adapter tuning, optimal transport, and fine-grained alignment between text tokens and image regions or video frames. Work also applies these methods to medical images, remote sensing, robotics, and multilingual settings, while developing benchmarks and diagnostics for compositional understanding and cross-modal correspondence. A central tension is that stronger semantic alignment can improve matching but may blur local details, amplify language bias, or weaken generalization across domains and tasks.

Latest papers

361 papers33 papers in the last four weeks, up 22% on the four weeks before. 0.2% of all new papers.

Language-guided Medical Image Segmentation with Target-informed Multi-level Contrastive Alignments
Dec 18, 2024Mingjian Li, Mingyuan Meng, Shuchang Ye +4Text-Guided Image Segmentation Medical Image Segmentation
a School of Computer Science, The University of Sydney, Sydney, Australia. · b Institute of Translational Medicine, Shanghai Jiao Tong University, Shanghai, China. · c Zhongguancun Academy & Zhongguancun Institute of Artificial Intelligence, Beijing, China. +1

Dec 18, 2024J/K move · Enter open · S save

Language-guided Medical Image Segmentation with Target-informed Multi-level Contrastive Alignments

Mingjian Li, Mingyuan Meng, Shuchang Ye, Mingye Zou+3

a School of Computer Science, The University of Sydney, Sydney, Australia. · b Institute of Translational Medicine, Shanghai Jiao Tong University, Shanghai, China. · c Zhongguancun Academy & Zhongguancun Institute of Artificial Intelligence, Beijing, China.+1

Text-Guided Image Segmentation Medical Image Segmentation Vision-Language Alignment Multimodal Contrastive Learning

Open paper AskPDF arXiv

Medical image segmentation is a fundamental task in numerous medical engineering applications. Recently, language-guided segmentation has shown promise in medical scenarios where textual clinical reports are readily available as semantic guidance. Clinical reports contain diagnostic information provided by clinicians, which can provide auxiliary textual semantics to guide segmentation. However, existing language-guided segmentation methods neglect the inherent pattern gaps between image and text modalities, resulting in sub-optimal visual-language integration. Contrastive learning is a well-recognized approach to align image-text patterns, but it has not been optimized for bridging the pattern gaps in medical language-guided segmentation that relies primarily on medical image details to characterize the underlying disease/targets. Current contrastive alignment techniques typically align high-level global semantics without involving low-level localized target information, and thus cannot deliver fine-grained textual guidance on crucial image details. In this study, we propose a Target-informed Multi-level Contrastive Alignment framework (TMCA) to bridge image-text pattern gaps for medical language-guided segmentation. TMCA enables target-informed image-text alignments and fine-grained textual guidance by introducing: (i) a target-sensitive semantic distance module that utilizes target information for more granular image-text alignment modeling, (ii) a multi-level contrastive alignment strategy that directs fine-grained textual guidance to multi-scale image details, and (iii) a language-guided target enhancement module that reinforces attention to critical image regions based on the aligned image-text patterns. Extensive experiments on four public benchmark datasets demonstrate that TMCA enabled superior performance over state-of-the-art language-guided medical image segmentation methods.