Multimodal Foundation Models

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 117% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 303

All topics

Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation
May 18, 2026Qianhao Yuan, Jie Lou, Xing Yu +4Multimodal Large Language Models On-Policy Distillation
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xiaohongshu Inc.
Lance: Unified Multimodal Modeling by Multi-Task Synergy
May 18, 2026Fengyi Fu, Mengqi Huang, Shaojin Wu +10Unified Multimodal Models Multi-Task Learning
Intelligent Creation Lab, ByteDance
RAVE: Re-Allocating Visual Attention in Large Multimodal Models
May 18, 2026Xi Leng, Xinhong Ma, Ziqiang Dong +4Visual Attention Self-Attention
Qwen Business Unit of Alibaba · The Chinese University of Hong Kong, Shenzhen · Beijing Institute of Technology
A More Word-like Image Tokenization for MLLMs
May 18, 2026Hyun Lee, Hyemin Jeong, Yejin Kim +4Image Tokenization Efficient VLM Inference
Seoul National University · Ewha Womans University
LatentUMM: Dual Latent Alignment for Unified Multimodal Models
May 18, 2026Yinyi Luo, Wenwen Wang, Hayes Bai +2Cross-Modal Learning Unified Multimodal Models
Carnegie Mellon University · William & Mary
Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
May 16, 2026Firoj Alam, Shammur Absar Chowdhury, Enamul Hoque PrinceCross-Modal Alignment Multimodal Large Language Models
Qatar Computing Research Institute, HBKU, Qatar · York University, Canada
TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokens
May 15, 2026Jianpeng Cheng, Xian Wu, Jiangfan Zhang +10Efficient Multimodal Inference Multimodal Embedding
1Meta AI
Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models
May 15, 2026Yishun Lu, Wes ArmourSecond-Order Optimization Multimodal Optimization
University of Oxford, Oxford, United Kingdom
DarkLLM: Learning Language-Driven Adversarial Attacks with Large Language Models
May 15, 2026Ye Sun, Xin Wang, Jiaming Zhang +7Adversarial Attacks on VLMs Adversarial Perturbations
Fudan University · Nanyang Technological University · Tongji University
Unlocking Dense Metric Depth Estimation in VLMs
May 15, 2026Hanxun Yu, Xuan Qu, Yuxin Wang +2Vision-Language Models Metric Depth Estimation
Zhejiang University · Tencent Hunyuan LLM · HKUST +1
Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Models
May 15, 2026Yujun Tong, Dongliang Chang, Zijin Yin +3Unified Multimodal Models Multimodal Reasoning
Beijing University of Posts and Telecommunications · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance
Pelican-Unify 1.0: A Unified Embodied Intelligence Model for Understanding, Reasoning, Imagination and Action
May 14, 2026Yi Zhang, Yinda Chen, Che Liu +26Action-Conditioned Video Generation Unified Multimodal Models
Beijing Innovation Center of Humanoid Robotics (X-Humanoid) WFM System Group
Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attribution
May 14, 2026Tian Qin, Junzhe Chen, Yuqing Shi +3Efficient VLM Inference Hallucination in Language Models
Tsinghua University · The University of Sydney · Stanford University +1
Unified Pix Token And Word Token Generative Language Model
May 13, 2026Haun Leung, ZiNan WangImage Tokenization Self-Supervised Pre-Training
Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs
May 13, 2026Trung Nguyen Quang, Yiming Gao, Fanyi Pu +3Multimodal Grounding Multimodal Large Language Models
Nanyang Technological University · LMMs-Lab Team · Johns Hopkins University
Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
May 13, 2026Qinwu Xu, Yifan Jiang, Haoyu RenMultimodal Grounding CoT Reasoning
Meta AI · UT Austin · Current independent researcher
GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embedding
May 13, 2026Mayank Nautiyal, Li Ju, Andreas Hellander +2Cross-Modal Learning Flow Matching
Department of Information Technology, Uppsala University, Uppsala, Sweden · SciLifeLab, Uppsala University, Uppsala, Sweden
When Vision Speaks for Sound
May 13, 2026Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu +6Audio-Language Model Evaluation Audio-Visual Understanding
dUniversity of California, Davis · pPrinceton University · wUniversity of Wisconsin–Madison +1
DiM\textsuperscript{3}: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging
May 13, 2026Zijing Wang, Mingyang Wang, Ercong Nie +6Multilingual Language Models Multimodal Large Language Models
Northeastern University, China · CIS, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Germany +1
SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning
May 13, 2026Truong Pham, Anay Majee, Rishabh IyerCross-Modal Learning Cross-Modal Alignment
The University of Texas at Dallas
SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture
May 12, 2026Haiwen Diao, Penghao Wu, Hanming Deng +55Vision-Language Models Unified Multimodal Models
Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models
May 12, 2026Yanting Miao, Yutao Sun, Dexin Wang +8Multimodal Large Language Models Latent Visual Reasoning
1Qwen Large Model Application Team, Alibaba · 3Zhejiang University · University of Waterloo +1
OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models
May 12, 2026Yuchen Deng, Zidang Cai, Hai-Tao Zheng +3Audio-Visual Understanding Efficient Multimodal Inference
Tsinghua Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory
Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
May 12, 2026Che Liu, Lichao Ma, Xiangyu Tony Zhang +4Cross-Modal Learning Multimodal Large Language Models
StepFun-Audio Team
20/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
May 12, 2026DatologyAI, :, Siddharth Joshi +32Training Data Curation Large Vision-Language Models
DatologyAI Team1
LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?
May 11, 2026Xueqi Cheng, Yushun DongEfficient Multimodal Inference Multimodal Large Language Models
Department of Computer Science Florida State University University
HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer
May 11, 2026Qi Cai, Jingwen Chen, Chengmin Gao +22Diffusion Transformer Text-Guided Image Editing
Qwen-Image-2.0 Technical Report
May 11, 2026Bing Zhao, Chenfei Wu, Deqing Li +72T2I Generation Compositional T2I Generation
Phoenix-VL 1.5 Medium Technical Report
May 11, 2026Team Phoenix, :, Arka Ray +30Multilingual Language Models Domain Adaptation
HTX × Mistral AI ∗
The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space
May 11, 2026Xia Hu, Zhenrui Yue, Brian Potetz +4VLM Evaluation Spatial Reasoning Benchmarks
Google DeepMind · Stanford University · Google Research

May 18, 2026, cs.CVJ/K move · Enter open · S save

Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation

Qianhao Yuan, Jie Lou, Xing Yu, Hongyu Lin+3

Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xiaohongshu Inc.

Multimodal Large Language Models On-Policy Distillation Multimodal Foundation Models Knowledge Distillation Fine-Grained Visual Perception Multimodal Understanding

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) still struggle with fine-grained visual understanding, where answers often depend on small but decisive evidence in the full image. We observe a regional-to-global perception gap: the same MLLM answers fine-grained questions more accurately when conditioned on evidence-centered crops than on the corresponding full images, suggesting that many failures stem from difficulty to focus on relevant evidence rather than insufficient local recognition ability. Motivated by this observation, we propose Vision-OPD (Vision On-Policy Distillation), a regional-to-global self-distillation framework that transfers the model's own privileged regional perception to its full-image policy. Vision-OPD instantiates two conditional policies from the same MLLM: a crop-conditioned teacher and a full-image-conditioned student. The student generates on-policy rollouts, and Vision-OPD minimizes token-level divergence between the teacher and student next-token distributions along these rollouts. This enables the model to internalize the benefit of visual zooming without external teacher models, ground-truth labels, reward verifiers, or inference-time tool use. Experiments on multiple fine-grained visual understanding benchmarks show that Vision-OPD models achieve competitive or superior performance against much larger open-source, closed-source, and "Thinking-with-Images" agentic models. The code is available at https://github.com/VisionOPD/Vision-OPD

Momentum

Related topics

Latest papers 303