Multimodal Foundation Models

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 117% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 303

All topics

TRACER: Persistent Regularization for Robust Multimodal Finetuning
May 28, 2026Hesam Asadollahzadeh, Feng Liu, Christopher Leckie +1VLM Robustness Neural Network Robustness
School of Computing and Information Systems (CIS), Faculty of Engineering and IT (FEIT), University of Melbourne, Australia.
From Pixels to Words -- Towards Native One-Vision Models at Scale
May 27, 2026Haiwen Diao, Jiahao Wang, Penghao Wu +18Vision-Language Models Video-Language Models
S-Lab, NTU · SenseTime Research · 3DLUT
OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibration
May 27, 2026Xinchen Zhang, Bowei Liu, Jiale Liu +7Multimodal Large Language Models Reinforcement Learning with Verifiable Rewards
Tsinghua University · Pennsylvania State University · University of Southern California +2
Bounded-Compute Multimodal Regression for Product-Rating Prediction
May 26, 2026William Leach, Ru He, Sizhuo Ma +4Vision-Language Models VLM Adaptation
Snap Inc.
Heterogeneous Parallelism for Multimodal Large Language Model Training
May 26, 2026Yashaswi Karnati, Kamran Jafari, Akash Mehra +10Communication-Efficient Distributed Training Multimodal Large Language Models
NVIDIA
Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini
May 26, 2026Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang +86Cross-Modal Learning Unified Multimodal Models
Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models
May 26, 2026Aaron Branson Cigres Li, Zhaowei Wang, Yu Zhao +9Vision-Language Models VLM Interpretability
1HKUST · University of Edinburgh · 4CUHK +2
DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding
May 26, 2026Peng Zhang, Guanghao Zhang, Wanggui He +10Multimodal Large Language Models Video Reasoning
Zhejiang University · Alibaba Group
OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation
May 26, 2026Yunze Liu, Chi-Hao Wu, Enmin Zhou +1Cross-Modal Learning Cross-Modal Knowledge Distillation
Memories.ai Research
FTibSuite: A Comprehensive Resource Suite for Tibetan Vision-Language Modeling
May 26, 2026Guixian Xu, Yide Liang, Zeli Su +5VLM Evaluation Vision-Language Models
Hainan International College, Minzu University of China · School of Information Engineering, Minzu University of China · Institute of Automation, Chinese Academy of Sciences +1
O-MARC: Omni Memory-Augmented Compression Distillation for Efficient Video Understanding
May 26, 2026Peiran Wu, Yunze Liu, Chi-Hao Wu +2Audio-Visual Understanding Efficient Multimodal Inference
University of Bristol · Memories.ai Research · University of Central Florida
OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants
May 26, 2026Xudong Lu, Xueying Li, Annan Wang +8Streaming Video Understanding Multimodal Foundation Models
CUHK MMLab · SJTU · NTU +3
Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
May 25, 2026Bonan Ding, Umair Nawaz, Ufaq Khan +5Cross-Modal Learning Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence · Tianjin University · Chongqing University +1
MAGIC: Multimodal Alignment & Grounding-aware Instruction Coreset for Vision-Language Models
May 25, 2026Shristi Das Biswas, Kaushik RoyVision-Language Models Coreset Selection
Purdue University
CMAP: Cross-Modal Adaptive Prompting for Multi-Domain Task-Incremental Learning
May 25, 2026Sriram MandalikaCross-Modal Learning VLM Adaptation
Hasso Plattner Institute Potsdam, Germany
MAIL++: Multi-Modal Bi-directional Agent Layer for Vision-Language Models
May 25, 2026Kaixiang Chen, Pengfei Fang, Hui XueCross-Modal Learning Cross-Modal Representation Learning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
May 25, 2026Zili Zhang, Chengxu Yang, Shenglong Zhang +8Deep Learning Optimization Multimodal Large Language Models
1Peking University · †Work done during internships at Xiaohongshu, Inc. · 2Independent Researcher +2
Toward Native Multimodal Modeling: A Roadmap
May 25, 2026Siyu An, Junru Lu, Junnan Dong +18Unified Multimodal Models Multimodal Generation
Tencent Youtu Lab
DIVA: Harnessing the Representation Divergence in Unified Multimodal Models for Mutual Reinforcement
May 25, 2026Renjie Lu, Xulong Zhang, Xiaoyang Qu +2Multimodal Disentangled Representation Learning Unified Multimodal Models
Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China · University of Science and Technology of China, Hefei, China.
Vision-Language Binding in In-Context Image Generation
May 23, 2026Chris Ge, Rohit Gandikota, Antonio Torralba +1Image Generation Mechanistic Interpretability
MIT CSAIL · Northeastern University
ETCHR: Editing To Clarify and Harness Reasoning
May 22, 2026Beichen Zhang, Yuhong Liu, Jinsong Li +3Tool Use in VLMs Multimodal Reasoning
The Chinese University of Hong Kong · Shanghai AI Laboratory · Shanghai Jiao Tong University +2
Cambrian-P: Pose-Grounded Video Understanding
May 21, 2026Jihan Yang, Zifan Zhao, Xichen Pan +5Vision-Language Models Camera Pose Estimation
New York University · UC Berkeley · Meta FAIR
Bernini: Latent Semantic Planning for Video Diffusion
May 21, 2026Bernini Team, Chenchen Liu, Junyi Chen +9Video Diffusion Models Unified Multimodal Models
Bernini Team, Bytedance · Bytedance
LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
May 21, 2026Yifan Dai, Zhenhua Wu, Bohan Zeng +18Cross-Modal Learning Audio-Visual Understanding
School of AI, Shanghai Jiao Tong University · 2Kling Team, Kuaishou Technology · 3Peking University +5
Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models
May 20, 2026Dong Chen, Fangyun Wei, Ziyu Wan +18Multimodal Foundation Models T2I Diffusion Models
Microsoft Lens Team
MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset
May 20, 2026Benjamin Aubin, Gonzalo Iñaki Quintana, Onur Tasar +4Training Data Curation T2I Generation
Jasper Research
Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach
May 20, 2026Herman Bergström, Aditya Mehrotra, Rahul G. KrishnanMultimodal Embedding Multimodal Classification
Chalmers University of Technology and University of Gothenburg · Vector Institute · University of Toronto
AffectVerse: Emotional World Models for Multimodal Affective Computing
May 19, 2026Bo Zhao, Fanghua Ye, Yixin Ji +3Cross-Modal Learning Multimodal Large Language Models
1Great Bay University · 2Tencent · 3Tsinghua University +1
Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models
May 19, 2026Tobias Braun, Jonas Henry Grebe, Hossein Shakibania +2Unified Multimodal Models Backdoor Attacks
TU Darmstadt & hessian.AI, Germany
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
May 18, 2026Paul Quinlan, Jeremy Levasseur, Qingguo Li +1Cross-Modal Learning Multivariate Time Series Forecasting
1InertialAI · Department of Electrical and Computer Engineering, Queen’s University · Department of Mechanical and Materials Engineering, Queen’s University

May 28, 2026, cs.LGJ/K move · Enter open · S save

TRACER: Persistent Regularization for Robust Multimodal Finetuning

Hesam Asadollahzadeh, Feng Liu, Christopher Leckie, Sarah M. Erfani

School of Computing and Information Systems (CIS), Faculty of Engineering and IT (FEIT), University of Melbourne, Australia.

VLM Robustness Neural Network Robustness OOD Generalization Self-Distillation Multimodal Foundation Models Multimodal Contrastive Learning+2

Open paper AskPDF arXiv

Mainstream strategies for finetuning pretrained multimodal models often degrade out-of-distribution (OOD) robustness, a phenomenon known as catastrophic forgetting. In this paper, we develop a theoretical framework for multimodal contrastive finetuning, yielding closed-form solutions and a geometric decomposition for each strategy. This framework shows that self-distillation is more effective than other regularization approaches to retain the knowledge of the pretrained model. Our analysis reveals a largely overlooked limitation: standard Exponential Moving Average (EMA) teachers, widely used in robust finetuning, suffer from collapse. To solve this, we prove that a Weighted Moving Average (WMA) teacher maintains a persistent regularizing force over finite horizons and yields bias-free convergence in the task subspace while preserving orthogonal knowledge. These insights motivate TRACER (Trajectory-Robust Anchoring for Contrastive Encoder Regularization), which combines contrastive learning with WMA-guided multi-perspective distillation. Extensive experiments on CLIP finetuning demonstrate consistent OOD accuracy and calibration gains across three backbone architectures, and comprehensive ablations confirm that TRACER is both principled and robust to hyperparameter choices. Code is available at https://github.com/HesamAsad/TRACER.

Momentum

Related topics

Latest papers 303