Multimodal Foundation Models

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 117% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 303

All topics

MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models
Jun 10, 2026Yuansheng Gao, Wenbin Xing, Jiahao Yuan +4Video-Language Models Multimodal Foundation Models
Zhejiang University · Sun Yat-sen University · East China Normal University
ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations
Jun 9, 2026Junke Wang, Xiao Wang, Jiacheng Pan +16Text-Guided Image Editing Unified Multimodal Models
Institute of Trustworthy Embodied AI, Fudan University
Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks
Jun 9, 2026Miaoxin Cai, Guanqun Wang, Wei Zhang +6Remote Sensing Image Understanding Multimodal Grounding
National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing (SBIIP), Beijing Institute of Technology, Beijing 100081, China · Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology, Beijing 100081, China · School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China +5
Kwai Keye-VL-2.0 Technical Report
Jun 9, 2026Kwai Keye Team, Bin Wen, Changyi Liu +50Temporal Video Grounding Long-Context Modeling
RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification
Jun 8, 2026Shakhrul Iman Siam, Tiantian Feng, Jiankun Zhang +2Healthcare Audio-Language Models
The Ohio State University · University of Southern California · University of Chicago
Next-Token Prediction Learns Generalisable Representations of Sleep Physiology
Jun 8, 2026Jonathan F. Carter, Lionel TarassenkoSleep Stage Classification Time Series Classification
Institute of Biomedical Engineering University of Oxford
AlloSpatial: Agentic Harness Framework for Spatial Reasoning in Foundation Models
Jun 8, 2026Shouwei Ruan, Bin Wang, Zhenyu Wu +5VLM Reasoning Multimodal Foundation Models
Institute of Artificial Intelligence, Beihang University · Huawei Noah’s Ark Lab · University of Science and Technology Beijing
Test-Time Scaling in Multimodal Foundation Models: A Comprehensive Survey of Generation and Reasoning
Jun 6, 2026Cong Wan, Ying He, Zhongzhan Huang +1Test-Time Scaling Multimodal Reasoning
Sun Yat-sen University
TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Models
Jun 4, 2026Ziwen Kan, Yishuo Chen, Kecheng Li +7Irregular Time-Series Modeling Missing-Modality Learning
Department of Computer Science, University of Central Florida · Department of Computer Science, University of North Carolina at Chapel Hill
Beyond Generative Decoding: Discriminative Hidden-State Readout from a Native Omni-Modal LLM for Multimodal Sentiment Analysis
Jun 4, 2026Bin Wen, Tien-Ping TanEfficient Multimodal Inference Sentiment Analysis
School of Computer Sciences, Universiti Sains Malaysia, Penang, Malaysia.
Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning
Jun 3, 2026Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh AfghahEfficient Multimodal Inference Multimodal ICL
Holcombe Department of Electrical and Computer Engineering, Clemson University, Clemson, United States.
Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models
Jun 3, 2026Shuwen Yu, Zhanxuan Hu, Yi Zhao +2Cross-Modal Representation Learning Vision Foundation Model Adaptation
Yunnan Normal University, Kunming, China · Kunming University of Science and Technology, Kunming, China
Frames2LoRA: Parametric Video Internalization for Vision-Language Models
Jun 3, 2026Manan Suri, Sarvesh Baskar, Dinesh ManochaVLM Adaptation Efficient VLM Inference
University of Maryland, College Park
Differentiable Efficient Operator Search
Jun 3, 2026Xiaohuan Pei, Jiyuan Zhang, Yuanfan Guo +4Efficient Multimodal Inference Multimodal Token Compression
1The University of Sydney · 2ByteDance · 3Shanghai Jiao Tong University +1
KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models
Jun 2, 2026Youqi Wu, Mohammad Jalali, Farzan FarniaKernel Methods Multimodal Foundation Models
Department of Computer Science and Engineering, The Chinese University of Hong Kong.
Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs
Jun 2, 2026Wei Ding, Yudong Zhang, Ruobing Xie +3VLM Evaluation Vision-Language Models
Tsinghua University Beijing, China · Tsinghua University, Tencent Beijing, China · University of Science and Technology Beijing Beijing, China
Investigating Adversarial Robustness of Multi-modal Large Language Models
Jun 2, 2026Hashmat Shadab Malik, Muzammal Naseer, Salman KhanAdversarial Training VLM Robustness
Mohamed Bin Zayed University of AI, UAE · Khalifa University, UAE · Australian National University, Australia
Attend to Anything: Foundation Model for Unified Human Attention Modeling
Jun 2, 2026Wenzhuo Zhao, Ronghao Xian, Keren Fu +1Visual Attention Unified Multimodal Models
College of Computer Science, Sichuan University, Chengdu, 610065, China · National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu, 610065, China
Cosmos 3: Omnimodal World Models for Physical AI
Jun 1, 2026NVIDIA, :, Aditi +293Unified Multimodal Models World Models
AdaCodec: A Predictive Visual Code for Video MLLMs
Jun 1, 2026Haowen Hou, Zhen Huang, Zheming Liang +8Predictive Coding Efficient VLM Inference
Shanghai Jiao Tong University · Shanghai Innovation Institute
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
May 31, 2026Wanlong Fang, Tianle Zhang, Wen Tao +1Multimodal Grounding Multimodal Large Language Models
College of Computing and Data Science, Nanyang Technological University, Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore · Centre of AI in Medicine (C-AIM), Nanyang Technological University, Singapore
MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models
May 30, 2026Ravil Mussabayev, Rustam MussabayevRepresentation Geometry Multimodal Large Language Models
Satbayev University
Zamba2-VL Technical Report
May 29, 2026Hassan Shapourian, Kasra Hejazi, Olabode M. Sule +1Vision-Language Models Efficient VLM Inference
Zyphra, San Francisco, CA
Representation Forcing for Bottleneck-Free Unified Multimodal Models
May 29, 2026Yuqing Wang, Zhijie Lin, Ceyuan Yang +10Unified Multimodal Models Conditional Image Generation
University of Hong Kong · ByteDance Seed · The Chinese University of Hong Kong +2
VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning
May 29, 2026Hengbo Xu, Shengjie Jin, Yanbiao Ma +1Efficient Multimodal Inference Visual Attention
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China.
"Înţelegi Româneşte?'' A Recipe for Romanian Vision-Language Models
May 29, 2026Mihai Masala, Marius Leordeanu, Mihai Dascalu +1VLM Evaluation Vision-Language Models
National University of Science and Technology POLITEHNICA Bucharest, Romania
GGT-100K: Generative Ground Truth for Generalizable Real-World Image Restoration
May 29, 2026Xiangtao Kong, Jixin Zhao, Lingchen Sun +2Synthetic Data Augmentation Image Restoration
The Hong Kong Polytechnic University · OPPO Research Institute
Your Multimodal Speech Model Says I Have a Face for Radio
May 28, 2026Maya K. Nachesa, Vlad Niculae, Vagrant GautamASR Evaluation Demographic Bias in ASR
Language Technology Lab University of Amsterdam · Heidelberg Institute for Theoretical Studies
Archon: A Unified Multimodal Model for Holistic Digital Human Generation
May 28, 2026Chong Bao, Shichen Liu, Lijun Yu +9Unified Multimodal Models Multimodal Pretraining
State Key Lab of CAD&CG, Zhejiang University · Google · The work was done during an internship at Google. +1
ReasonLight: A Multimodal Foundation Model-Enhanced Reinforcement Learning Framework for Zero-Shot Traffic Signal Control
May 28, 2026Aoyu Pang, Maonan Wang, Yuejiao Xie +3RL Control Intelligent Transportation Systems
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong · Shanghai AI Laboratory, Shanghai, China +1

Jun 10, 2026, cs.CVJ/K move · Enter open · S save

MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models

Yuansheng Gao, Wenbin Xing, Jiahao Yuan, Kaiwen Zhou+3

Zhejiang University · Sun Yat-sen University · East China Normal University

Video-Language Models Multimodal Foundation Models VLM Hallucination Mitigation Multimodal Understanding

Open paper AskPDF arXiv

Video Large Multimodal Models have achieved remarkable progress in video understanding, yet they remain prone to hallucinations, where generated responses are not faithfully supported by the input video. In this paper, we propose MultiToP, a multimodal-context-aware visual token patching framework that mitigates hallucinations by refining unreliable visual tokens before language generation. MultiToP introduces a lightweight Visual Token Patcher to predict token-level replacement distributions and selectively substitute unreliable visual tokens with a dynamic global patch token. To train the patcher effectively, we further propose information-guided rank calibration, which uses answer-conditioned frame-level information cues derived from the backbone to guide token replacement. Combined with ground-truth answer supervision and sparsity regularization, MultiToP enables localized visual evidence refinement without modifying the original model. Extensive experiments demonstrate that MultiToP effectively reduces hallucinations on Vript-HAL with negligible inference overhead, improving the F1 scores of Qwen3-VL-4B-Instruct by 50.60% over the vanilla model. Meanwhile, MultiToP preserves general video understanding ability, yielding an 18.58% relative accuracy gain on ActivityNet-QA for Video-LLaVA-7B.

Momentum

Related topics

Latest papers 303