Vision-Language Pretraining

Ask about this topic

Overview

Vision-language pretraining teaches models to connect visual content with language, supporting image-text retrieval, zero-shot classification, question answering, and applications such as medical image analysis. Common approaches include contrastive image-text alignment, caption generation, masked or predictive feature learning, and region-level grounding, trained on web captions, curated datasets, or domain-specific reports. Researchers evaluate transfer on classification and retrieval benchmarks such as ImageNet and COCO, as well as on dense perception, visual question answering, documents, and clinical tasks. A central tension is that global alignment scales well but can miss fine details and composition, while local or generative objectives offer richer supervision but vary in benefit across tasks and settings.

Latest papers

CardsList

80 papers7 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

  1. GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretraining

    Mar 25, 2026Deen Dayal Mohan, Hossein Souri, Vitali Petsiuk +4Vision-Language PretrainingVLM Distillation

    AI Center – Mountain View, Samsung Electronics

  2. IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment

    Mar 20, 2026Simone Magistri, Dipam Goswami, Marco Mistretta +3Latent Space AlignmentModality Gap in VLMs

    Media Integration and Communication Center (MICC), University of Florence, Italy · Department of Computer Science, Universitat Aut`onoma de Barcelona, Spain · Computer Vision Center, Barcelona, Spain +1

  3. FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptions

    Mar 18, 2026Peisen Zhao, Xiaopeng Zhang, Mingxing Xu +10Fine-Grained Visual PerceptionVision-Language Pretraining

    Huawei Inc.

  4. RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning

    Mar 10, 2026Tzu-Heng Huang, Sirajul Salekin, Javier Movellan +2Image CaptioningRL for VLMs

    Work done during an internship at Apple · University of Wisconsin—Madison · Apple

  5. VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs

    Mar 10, 2026Xiyao Wang, Xiaoyu Tan, Yang Dai +3Medical Image Representation LearningVision-Language Pretraining

    Shanghai University of Engineering Science, China · National University of Singapore, Singapore · Case Western Reserve University, United States

  6. MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestration

    Mar 10, 2026Chenran Zhang, Ruiqi Wu, Tao Zhou +1Medical VLMsVision-Language Pretraining

    School of Computer Science and Engineering, Southeast University, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, China

  7. Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders

    Mar 6, 2026Boqiang Zhang, Lei Ke, Ruihan Yang +5Vision-Language ModelsVision-Language Pretraining

  8. Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languages

    Mar 6, 2026Shaharukh Khan, Ali Faraz, Abhinav Ravi +6Vision-Language PretrainingMultimodal Pretraining

    Krutrim AI, Bangalore, India

  9. Location-Aware Pretraining for Medical Difference Visual Question Answering

    Mar 5, 2026Denis Musinguzi, Caren Han, Prasenjit MitraMedical VQAVision-Language Pretraining

    Department of Electrical and Computer Engineering, Carnegie Mellon University, Kigali, Rwanda · University of Melbourne, Melbourne, Australia.

  10. Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interaction

    Mar 5, 2026Yuanbo Li, Tianyang Xu, Cong Hu +3Adversarial Attacks on VLMsTransfer-Based Adversarial Attacks

    School of Artificial Intelligence and Computer Science, Jiangnan University · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey

  11. ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining

    Mar 3, 2026Hanpeng Liu, Yaqian Li, Zidan Wang +6Cross-Modal LearningCross-Modal Alignment

    School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China · Li Auto Inc. · Institute of AI for Industries, Chinese Academy of Sciences

  12. 3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection

    Feb 27, 2026Haowen Zhu, Ning Yin, Xiaogen ZhouMedical VLMsVision-Language Pretraining

    School of Computer Science and Engineering, Southeast University, China · Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China · School of Electronic, Electrical Engineering and Physics, Fujian University of Technology

  13. CLIP Is Shortsighted: Paying Attention Beyond the First Sentence

    Feb 25, 2026Marc-Antoine Lavoie, Anas Mahmoud, Aldo Zaimi +2Position Bias in VLMsVision-Language Alignment

    University of Toronto Robotics Institute · Mila - Quebec AI Institute

  14. MedTri: A Platform for Structured Medical Report Normalization to Enhance Vision-Language Pretraining

    Feb 25, 2026Yuetan Chu, Xinhua Ma, Xinran Jin +2Medical VLMsVision-Language Pretraining

    King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Faculty of Computing, Harbin, China.

  15. SigVLP: Sigmoid Volume-Language Pre-Training for Self-Supervised CT-Volume Adaptive Representation Learning

    Feb 25, 2026Jiayi Wang, Hadrien Reynaud, Ibrahim Ethem Hamamci +4Medical VLMsMedical Image Representation Learning

    Friedrich-Alexander University Erlangen-Nürnberg, Germany · Department of Quantitative Biomedicine, University of Zurich, Switzerland · ETH AI Center, ETH Zurich, Switzerland +2

  16. Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models

    Feb 19, 2026Dhruba Ghosh, Yuhui Zhang, Ludwig SchmidtFine-Grained Visual PerceptionVLM Evaluation

    Stanford University

  17. Xray-Visual Models: Scaling Vision models on Industry Scale Data

    Feb 18, 2026Shlok Mishra, Tsung-Yu Lin, Linda Wang +24Vision Foundation ModelsMultimodal Pretraining

    Meta-AI · MIT (work done while at Meta AI) · University of Maryland (work done while at Meta AI)

  18. How to Train Your Long-Context Visual Document Model

    Feb 16, 2026Austin VeselkaVisual Question AnsweringVision-Language Models

    LightOn

  19. MTV: Revisiting Multi-Task Visual Representation Learning

    Jan 20, 2026Shangzhe Di, Zhonghua Zhai, Weidi XieVisual Representation LearningSelf-Supervised Visual Representation Learning

    SAI, Shanghai Jiao Tong University · ByteDance Seed

  20. CLIMP: Contrastive Language-Image Mamba Pretraining

    Jan 11, 2026Nimrod Shabtay, Itamar Zimerman, Eli Schwartz +1Vision-Language PretrainingVLM Robustness

    Tel-Aviv University · IBM Research