Multimodal Foundation Models

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 117% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 303

All topics

MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models
Jun 10, 2026Yuansheng Gao, Wenbin Xing, Jiahao Yuan +4Video-Language Models Multimodal Foundation Models
Zhejiang University · Sun Yat-sen University · East China Normal University
ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations
Jun 9, 2026Junke Wang, Xiao Wang, Jiacheng Pan +16Text-Guided Image Editing Unified Multimodal Models
Institute of Trustworthy Embodied AI, Fudan University
Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks
Jun 9, 2026Miaoxin Cai, Guanqun Wang, Wei Zhang +6Remote Sensing Image Understanding Multimodal Grounding
National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing (SBIIP), Beijing Institute of Technology, Beijing 100081, China · Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology, Beijing 100081, China · School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China +5
Kwai Keye-VL-2.0 Technical Report
Jun 9, 2026Kwai Keye Team, Bin Wen, Changyi Liu +50Temporal Video Grounding Long-Context Modeling
RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification
Jun 8, 2026Shakhrul Iman Siam, Tiantian Feng, Jiankun Zhang +2Healthcare Audio-Language Models
The Ohio State University · University of Southern California · University of Chicago
Next-Token Prediction Learns Generalisable Representations of Sleep Physiology
Jun 8, 2026Jonathan F. Carter, Lionel TarassenkoSleep Stage Classification Time Series Classification
Institute of Biomedical Engineering University of Oxford
AlloSpatial: Agentic Harness Framework for Spatial Reasoning in Foundation Models
Jun 8, 2026Shouwei Ruan, Bin Wang, Zhenyu Wu +5VLM Reasoning Multimodal Foundation Models
Institute of Artificial Intelligence, Beihang University · Huawei Noah’s Ark Lab · University of Science and Technology Beijing
Test-Time Scaling in Multimodal Foundation Models: A Comprehensive Survey of Generation and Reasoning
Jun 6, 2026Cong Wan, Ying He, Zhongzhan Huang +1Test-Time Scaling Multimodal Reasoning
Sun Yat-sen University
TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Models
Jun 4, 2026Ziwen Kan, Yishuo Chen, Kecheng Li +7Irregular Time-Series Modeling Missing-Modality Learning
Department of Computer Science, University of Central Florida · Department of Computer Science, University of North Carolina at Chapel Hill
Beyond Generative Decoding: Discriminative Hidden-State Readout from a Native Omni-Modal LLM for Multimodal Sentiment Analysis
Jun 4, 2026Bin Wen, Tien-Ping TanEfficient Multimodal Inference Sentiment Analysis
School of Computer Sciences, Universiti Sains Malaysia, Penang, Malaysia.
Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning
Jun 3, 2026Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh AfghahEfficient Multimodal Inference Multimodal ICL
Holcombe Department of Electrical and Computer Engineering, Clemson University, Clemson, United States.
Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models
Jun 3, 2026Shuwen Yu, Zhanxuan Hu, Yi Zhao +2Cross-Modal Representation Learning Vision Foundation Model Adaptation
Yunnan Normal University, Kunming, China · Kunming University of Science and Technology, Kunming, China
Frames2LoRA: Parametric Video Internalization for Vision-Language Models
Jun 3, 2026Manan Suri, Sarvesh Baskar, Dinesh ManochaVLM Adaptation Efficient VLM Inference
University of Maryland, College Park
Differentiable Efficient Operator Search
Jun 3, 2026Xiaohuan Pei, Jiyuan Zhang, Yuanfan Guo +4Efficient Multimodal Inference Multimodal Token Compression
1The University of Sydney · 2ByteDance · 3Shanghai Jiao Tong University +1
KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models
Jun 2, 2026Youqi Wu, Mohammad Jalali, Farzan FarniaKernel Methods Multimodal Foundation Models
Department of Computer Science and Engineering, The Chinese University of Hong Kong.
Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs
Jun 2, 2026Wei Ding, Yudong Zhang, Ruobing Xie +3VLM Evaluation Vision-Language Models
Tsinghua University Beijing, China · Tsinghua University, Tencent Beijing, China · University of Science and Technology Beijing Beijing, China
Investigating Adversarial Robustness of Multi-modal Large Language Models
Jun 2, 2026Hashmat Shadab Malik, Muzammal Naseer, Salman KhanAdversarial Training VLM Robustness
Mohamed Bin Zayed University of AI, UAE · Khalifa University, UAE · Australian National University, Australia
Attend to Anything: Foundation Model for Unified Human Attention Modeling
Jun 2, 2026Wenzhuo Zhao, Ronghao Xian, Keren Fu +1Visual Attention Unified Multimodal Models
College of Computer Science, Sichuan University, Chengdu, 610065, China · National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu, 610065, China
Cosmos 3: Omnimodal World Models for Physical AI
Jun 1, 2026NVIDIA, :, Aditi +293Unified Multimodal Models World Models
AdaCodec: A Predictive Visual Code for Video MLLMs
Jun 1, 2026Haowen Hou, Zhen Huang, Zheming Liang +8Predictive Coding Efficient VLM Inference
Shanghai Jiao Tong University · Shanghai Innovation Institute
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
May 31, 2026Wanlong Fang, Tianle Zhang, Wen Tao +1Multimodal Grounding Multimodal Large Language Models
College of Computing and Data Science, Nanyang Technological University, Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore · Centre of AI in Medicine (C-AIM), Nanyang Technological University, Singapore
MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models
May 30, 2026Ravil Mussabayev, Rustam MussabayevRepresentation Geometry Multimodal Large Language Models
Satbayev University
Zamba2-VL Technical Report
May 29, 2026Hassan Shapourian, Kasra Hejazi, Olabode M. Sule +1Vision-Language Models Efficient VLM Inference
Zyphra, San Francisco, CA
Representation Forcing for Bottleneck-Free Unified Multimodal Models
May 29, 2026Yuqing Wang, Zhijie Lin, Ceyuan Yang +10Unified Multimodal Models Conditional Image Generation
University of Hong Kong · ByteDance Seed · The Chinese University of Hong Kong +2
VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning
May 29, 2026Hengbo Xu, Shengjie Jin, Yanbiao Ma +1Efficient Multimodal Inference Visual Attention
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China.
"Înţelegi Româneşte?'' A Recipe for Romanian Vision-Language Models
May 29, 2026Mihai Masala, Marius Leordeanu, Mihai Dascalu +1VLM Evaluation Vision-Language Models
National University of Science and Technology POLITEHNICA Bucharest, Romania
GGT-100K: Generative Ground Truth for Generalizable Real-World Image Restoration
May 29, 2026Xiangtao Kong, Jixin Zhao, Lingchen Sun +2Synthetic Data Augmentation Image Restoration
The Hong Kong Polytechnic University · OPPO Research Institute
Your Multimodal Speech Model Says I Have a Face for Radio
May 28, 2026Maya K. Nachesa, Vlad Niculae, Vagrant GautamASR Evaluation Demographic Bias in ASR
Language Technology Lab University of Amsterdam · Heidelberg Institute for Theoretical Studies
Archon: A Unified Multimodal Model for Holistic Digital Human Generation
May 28, 2026Chong Bao, Shichen Liu, Lijun Yu +9Unified Multimodal Models Multimodal Pretraining
State Key Lab of CAD&CG, Zhejiang University · Google · The work was done during an internship at Google. +1
ReasonLight: A Multimodal Foundation Model-Enhanced Reinforcement Learning Framework for Zero-Shot Traffic Signal Control
May 28, 2026Aoyu Pang, Maonan Wang, Yuejiao Xie +3RL Control Intelligent Transportation Systems
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong · Shanghai AI Laboratory, Shanghai, China +1
TRACER: Persistent Regularization for Robust Multimodal Finetuning
May 28, 2026Hesam Asadollahzadeh, Feng Liu, Christopher Leckie +1VLM Robustness Neural Network Robustness
School of Computing and Information Systems (CIS), Faculty of Engineering and IT (FEIT), University of Melbourne, Australia.
From Pixels to Words -- Towards Native One-Vision Models at Scale
May 27, 2026Haiwen Diao, Jiahao Wang, Penghao Wu +18Vision-Language Models Video-Language Models
S-Lab, NTU · SenseTime Research · 3DLUT
OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibration
May 27, 2026Xinchen Zhang, Bowei Liu, Jiale Liu +7Multimodal Large Language Models Reinforcement Learning with Verifiable Rewards
Tsinghua University · Pennsylvania State University · University of Southern California +2
Bounded-Compute Multimodal Regression for Product-Rating Prediction
May 26, 2026William Leach, Ru He, Sizhuo Ma +4Vision-Language Models VLM Adaptation
Snap Inc.
Heterogeneous Parallelism for Multimodal Large Language Model Training
May 26, 2026Yashaswi Karnati, Kamran Jafari, Akash Mehra +10Communication-Efficient Distributed Training Multimodal Large Language Models
NVIDIA
Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini
May 26, 2026Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang +86Cross-Modal Learning Unified Multimodal Models
Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models
May 26, 2026Aaron Branson Cigres Li, Zhaowei Wang, Yu Zhao +9Vision-Language Models VLM Interpretability
1HKUST · University of Edinburgh · 4CUHK +2
DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding
May 26, 2026Peng Zhang, Guanghao Zhang, Wanggui He +10Multimodal Large Language Models Video Reasoning
Zhejiang University · Alibaba Group
OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation
May 26, 2026Yunze Liu, Chi-Hao Wu, Enmin Zhou +1Cross-Modal Learning Cross-Modal Knowledge Distillation
Memories.ai Research
FTibSuite: A Comprehensive Resource Suite for Tibetan Vision-Language Modeling
May 26, 2026Guixian Xu, Yide Liang, Zeli Su +5VLM Evaluation Vision-Language Models
Hainan International College, Minzu University of China · School of Information Engineering, Minzu University of China · Institute of Automation, Chinese Academy of Sciences +1
O-MARC: Omni Memory-Augmented Compression Distillation for Efficient Video Understanding
May 26, 2026Peiran Wu, Yunze Liu, Chi-Hao Wu +2Audio-Visual Understanding Efficient Multimodal Inference
University of Bristol · Memories.ai Research · University of Central Florida
OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants
May 26, 2026Xudong Lu, Xueying Li, Annan Wang +8Streaming Video Understanding Multimodal Foundation Models
CUHK MMLab · SJTU · NTU +3
Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
May 25, 2026Bonan Ding, Umair Nawaz, Ufaq Khan +5Cross-Modal Learning Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence · Tianjin University · Chongqing University +1
MAGIC: Multimodal Alignment & Grounding-aware Instruction Coreset for Vision-Language Models
May 25, 2026Shristi Das Biswas, Kaushik RoyVision-Language Models Coreset Selection
Purdue University
CMAP: Cross-Modal Adaptive Prompting for Multi-Domain Task-Incremental Learning
May 25, 2026Sriram MandalikaCross-Modal Learning VLM Adaptation
Hasso Plattner Institute Potsdam, Germany
MAIL++: Multi-Modal Bi-directional Agent Layer for Vision-Language Models
May 25, 2026Kaixiang Chen, Pengfei Fang, Hui XueCross-Modal Learning Cross-Modal Representation Learning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
May 25, 2026Zili Zhang, Chengxu Yang, Shenglong Zhang +8Deep Learning Optimization Multimodal Large Language Models
1Peking University · †Work done during internships at Xiaohongshu, Inc. · 2Independent Researcher +2
Toward Native Multimodal Modeling: A Roadmap
May 25, 2026Siyu An, Junru Lu, Junnan Dong +18Unified Multimodal Models Multimodal Generation
Tencent Youtu Lab
DIVA: Harnessing the Representation Divergence in Unified Multimodal Models for Mutual Reinforcement
May 25, 2026Renjie Lu, Xulong Zhang, Xiaoyang Qu +2Multimodal Disentangled Representation Learning Unified Multimodal Models
Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China · University of Science and Technology of China, Hefei, China.
Vision-Language Binding in In-Context Image Generation
May 23, 2026Chris Ge, Rohit Gandikota, Antonio Torralba +1Image Generation Mechanistic Interpretability
MIT CSAIL · Northeastern University
ETCHR: Editing To Clarify and Harness Reasoning
May 22, 2026Beichen Zhang, Yuhong Liu, Jinsong Li +3Tool Use in VLMs Multimodal Reasoning
The Chinese University of Hong Kong · Shanghai AI Laboratory · Shanghai Jiao Tong University +2
Cambrian-P: Pose-Grounded Video Understanding
May 21, 2026Jihan Yang, Zifan Zhao, Xichen Pan +5Vision-Language Models Camera Pose Estimation
New York University · UC Berkeley · Meta FAIR
Bernini: Latent Semantic Planning for Video Diffusion
May 21, 2026Bernini Team, Chenchen Liu, Junyi Chen +9Video Diffusion Models Unified Multimodal Models
Bernini Team, Bytedance · Bytedance
LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
May 21, 2026Yifan Dai, Zhenhua Wu, Bohan Zeng +18Cross-Modal Learning Audio-Visual Understanding
School of AI, Shanghai Jiao Tong University · 2Kling Team, Kuaishou Technology · 3Peking University +5
Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models
May 20, 2026Dong Chen, Fangyun Wei, Ziyu Wan +18Multimodal Foundation Models T2I Diffusion Models
Microsoft Lens Team
MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset
May 20, 2026Benjamin Aubin, Gonzalo Iñaki Quintana, Onur Tasar +4Training Data Curation T2I Generation
Jasper Research
Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach
May 20, 2026Herman Bergström, Aditya Mehrotra, Rahul G. KrishnanMultimodal Embedding Multimodal Classification
Chalmers University of Technology and University of Gothenburg · Vector Institute · University of Toronto
AffectVerse: Emotional World Models for Multimodal Affective Computing
May 19, 2026Bo Zhao, Fanghua Ye, Yixin Ji +3Cross-Modal Learning Multimodal Large Language Models
1Great Bay University · 2Tencent · 3Tsinghua University +1
Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models
May 19, 2026Tobias Braun, Jonas Henry Grebe, Hossein Shakibania +2Unified Multimodal Models Backdoor Attacks
TU Darmstadt & hessian.AI, Germany
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
May 18, 2026Paul Quinlan, Jeremy Levasseur, Qingguo Li +1Cross-Modal Learning Multivariate Time Series Forecasting
1InertialAI · Department of Electrical and Computer Engineering, Queen’s University · Department of Mechanical and Materials Engineering, Queen’s University

Jun 10, 2026, cs.CVJ/K move · Enter open · S save

MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models

Yuansheng Gao, Wenbin Xing, Jiahao Yuan, Kaiwen Zhou+3

Zhejiang University · Sun Yat-sen University · East China Normal University

Video-Language Models Multimodal Foundation Models VLM Hallucination Mitigation Multimodal Understanding

Open paper AskPDF arXiv

Video Large Multimodal Models have achieved remarkable progress in video understanding, yet they remain prone to hallucinations, where generated responses are not faithfully supported by the input video. In this paper, we propose MultiToP, a multimodal-context-aware visual token patching framework that mitigates hallucinations by refining unreliable visual tokens before language generation. MultiToP introduces a lightweight Visual Token Patcher to predict token-level replacement distributions and selectively substitute unreliable visual tokens with a dynamic global patch token. To train the patcher effectively, we further propose information-guided rank calibration, which uses answer-conditioned frame-level information cues derived from the backbone to guide token replacement. Combined with ground-truth answer supervision and sparsity regularization, MultiToP enables localized visual evidence refinement without modifying the original model. Extensive experiments demonstrate that MultiToP effectively reduces hallucinations on Vript-HAL with negligible inference overhead, improving the F1 scores of Qwen3-VL-4B-Instruct by 50.60% over the vanilla model. Meanwhile, MultiToP preserves general video understanding ability, yielding an 18.58% relative accuracy gain on ActivityNet-QA for Video-LLaVA-7B.

Momentum

Related topics

Latest papers 303