Vision-Language Pretraining

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 50

All topics

What Does the Caption Really Say? Counterfactual Phrase Intervention for Compositional Data Selection in Vision-Language Pretraining
May 21, 2026Hyejin Go, Semi Lee, Hyesong ChoiTraining Data Selection Vision-Language Pretraining
Soongsil University Seoul, Republic of Korea
Unified Pix Token And Word Token Generative Language Model
May 13, 2026Haun Leung, ZiNan WangImage Tokenization Self-Supervised Pre-Training
Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context
May 13, 2026Zhaowei Wang, Lishu Luo, Haodong Duan +9Visual Question Answering Vision-Language Models
1CSE Department, HKUST · 2ByteDance Seed · *Work done at ByteDance Seed
CA-GCL: Cross-Anatomy Global-Local Contrastive Learning for Robust 3D Medical Image Understanding
May 13, 2026Hanwen Zhang, Yao Liu, Die Dai +4VLM Robustness Medical Image Anomaly Detection
University of Electronic Science and Technology of China, Chengdu, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
20/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
May 12, 2026DatologyAI, :, Siddharth Joshi +32Training Data Curation Large Vision-Language Models
DatologyAI Team1
Birds of a Feather Flock Together: Background-Invariant Representations via Linear Structure in VLMs
May 11, 2026Youssef Zaazou, Mark ThomasVLM Robustness Vision-Language Models
Independent Researcher
DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistency
May 7, 2026Shuyang Jiang, Nan Yu, Yiming Zhang +2Contrastive Learning Vision-Language Alignment
University of California, Los Angeles · Aimaikj · HFIPS, Chinese Academy of Sciences +2
HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
May 6, 2026Esra Dönmez, Pascal Tilli, Hsiu-Yu Yang +2Cross-Modal Learning Image Matching
Institute for Natural Language Processing, University of Stuttgart
Text-Conditional JEPA for Learning Semantically Rich Visual Representations
May 5, 2026Chen Huang, Xianhang Li, Vimal Thilak +2Joint-Embedding Predictive Architecture Predictive Representation Learning
Apple, Cupertino, United States
Scientific Domain Knowledge Improves Vision-Language Fundus Models
May 4, 2026Verena Jasmin Hallitschke, Carsten Eickhoff, Philipp BerensHealthcare Medical Diagnosis
1Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany · Institute for Applied Medical Informatics, University of Tübingen, Tübingen, Germany · 2Tübingen AI Center, Tübingen, Germany
Let ViT Speak: Generative Language-Image Pre-training
May 1, 2026Yan Fang, Mengcheng Lan, Zilong Huang +7Cross-Modal Learning Vision Transformer
1Beijing Jiaotong University · 2ByteDance · 3Nanyang Technological University
Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training
Apr 30, 2026Mingliang Liang, Zhuoran Liu, Arjen P. de Vries +1Long-Tail Learning Vision-Language Pretraining
Institute for Computing and Information Sciences, Radboud University, Nijmegen, the Netherlands · University of Amsterdam, Amsterdam, the Netherlands · Radboud University
Probing CLIP's Comprehension of 360-Degree Textual and Visual Semantics
Apr 27, 2026Hai Wang, Xiaochen Yang, Mingzhi Dong +1VLM Evaluation VLM Adaptation
Department of Statistical Science, University College London, UK · School of Mathematics and Statistics, University of Glasgow, UK · Department of Computer Science, University of Bath, UK.
PokeVLA: Empowering Pocket-Sized Vision-Language-Action Model with Comprehensive World Knowledge Guidance
Apr 22, 2026Yupeng Zheng, Xiang Li, Songen Gu +12Robot Foundation Models Efficient VLA Models
CASIA · TARS Robotics · Tsinghua University +3
Multilingual Training and Evaluation Resources for Vision-Language Models
Apr 20, 2026Daniela Baiamonte, Elena Fano, Matteo Gabburo +3VLM Evaluation Vision-Language Models
Aithlas
ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining
Mar 3, 2026Hanpeng Liu, Zidan Wang, Shuoxi Zhang +6Cross-Modal Learning Cross-Modal Alignment
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China · Li Auto Inc.
Xray-Visual Models: Scaling Vision models on Industry Scale Data
Feb 18, 2026Shlok Mishra, Tsung-Yu Lin, Linda Wang +24Vision Foundation Models Multimodal Pretraining
Meta-AI · MIT (work done while at Meta AI) · University of Maryland (work done while at Meta AI)
How to Train Your Long-Context Visual Document Model
Feb 16, 2026Austin VeselkaVisual Question Answering Vision-Language Models
LightOn
MTV: Revisiting Multi-Task Visual Representation Learning
Jan 20, 2026Shangzhe Di, Zhonghua Zhai, Weidi XieVisual Representation Learning Self-Supervised Visual Representation Learning
SAI, Shanghai Jiao Tong University · ByteDance Seed
CLIMP: Contrastive Language-Image Mamba Pretraining
Jan 11, 2026Nimrod Shabtay, Itamar Zimerman, Eli Schwartz +1Cross-Modal Learning VLM Robustness
Tel-Aviv University · IBM Research

May 21, 2026, cs.CVJ/K move · Enter open · S save

What Does the Caption Really Say? Counterfactual Phrase Intervention for Compositional Data Selection in Vision-Language Pretraining

Hyejin Go, Semi Lee, Hyesong Choi

Soongsil University Seoul, Republic of Korea

Training Data Selection Vision-Language Pretraining Compositional Generalization

Open paper AskPDF arXiv

CLIP-style contrastive pretraining typically curates web-scale image-text pairs using sample-level filtering signals, often based on pair-level alignment. We show that this signal saturates: once coarse mismatches are removed, stricter global filtering no longer tracks the compositional supervision provided by the retained captions. The reason is structural - a global score conflates whether a pair is broadly plausible with whether the individual object, attribute, and relation phrases inside the caption materially support the image-text match. The latter is what compositional generalization demands, yet pair-level filters are blind to it. We address this with Counterfactual Phrase Intervention (CPI), a phrase-level curation framework that converts controlled nonce-token substitutions into image-conditioned phrase-sensitivity scores. CPI uses global alignment only for coarse mismatch removal, then ranks the surviving pool by whether caption phrases measurably affect the image-text score under controlled substitution. We frame CPI as a first-order phrase-sensitivity signal rather than a grounding or identification result, and evaluate it at CC3M scale. Ranking by this signal yields a 50%-data subset that improves VL-CheckList-VG Relation by +1.91 over the full-data baseline and +1.00 over alignment-only filtering at matched budget, while improving SugarCrepe overall and preserving general transfer. CPI is loss-orthogonal: applied unchanged to NegCLIP, it further improves VL-CheckList-VG Relation by +3.84, with additional CE-CLIP gains in the main text.

Momentum

Related topics

Latest papers 50