Multimodal Understanding

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, up 233% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 413

All topics

Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs
Apr 16, 2026Ziyang Luo, Nian Liu, Junwei HanMultimodal Large Language Models Multimodal Reasoning
Northwestern Polytechnical University Xi’an, Shaanxi, China
Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs
Apr 13, 2026Kaizhen Tan, Yang Feng, Heqing Du +3VLM Evaluation Confidence Estimation in Language Models
Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA
Unified Multimodal Uncertain Inference
Apr 9, 2026Dengjia Zhang, Alexander Martin, William Jurayj +3Probability Calibration Multimodal Reasoning
Johns Hopkins University · Human Language Technology Center of Excellence
MG $^2$ -RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation
Apr 4, 2026Sijun Dai, Qiang Huang, Xiaoxing You +1Cross-Modal Learning Multimodal RAG
School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Computer Science, Hangzhou Dianzi University
Internalized Reasoning for Long-Context Visual Document Understanding
Mar 31, 2026Austin VeselkaLong-Context Modeling VLM Reasoning
LightOn
AMIGO: Agentic Multi-Image Grounding Oracle Benchmark
Mar 30, 2026Min Wang, Ata MahjoubfarVLM Evaluation Visual Question Answering
Target Corporation
BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QA
Mar 30, 2026Taeyun Roh, Suhyeong Park, Dongyoung Lee +4Vision-Language Models Multimodal QA
Korea University · AIGEN Sciences · KAIST +1
UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy
Mar 25, 2026Yicheng Xu, Jiangning Zhang, Zhucun Xue +5Multimodal ICL Unified Multimodal Models
1Zhejiang University, China. · 2Shanghai Jiaotong University, China. · 3National University of Singapore, Singapore. +1
UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation
Mar 23, 2026Ziyi Wang, Xinshun Wang, Shuang Chen +2Cross-Modal Learning Unified Multimodal Models
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China · Donghua University, China · School of Automation Science and Engineering, South China University of Technology
HumanOmni-Speaker: Identifying Who said What and When
Mar 23, 2026Detao Bai, Zhiheng Ma, Xihan WeiAudio-Visual Understanding Visual Speaker Recognition
Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology · Guangdong Provincial Key Laboratory of Computility Microelectronics +1
ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models
Mar 19, 2026Thomas De Min, Subhankar Roy, Stéphane Lathuilière +2Multimodal Large Language Models AI Agent Benchmarks
University of Trento · University of Bergamo · Inria, Univ. Grenoble Alpes, CNRS, LJK +1
HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering
Mar 19, 2026Dan Ben-Ami, Gabriele Serussi, Kobi Cohen +1Multimodal Large Language Models Multimodal QA
INSIGHT Lab, Ben-Gurion University of the Negev, Israel · Ben-Gurion University of the Negev, Israel
Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
Mar 6, 2026Lijiang Li, Zuwei Long, Yunhang Shen +6Masked Diffusion Models Unified Multimodal Models
Nanjing University · Tencent Youtu Lab · CASIA
Dual-Modality Multi-Stage Adversarial Safety Training: Robustifying Multimodal Web Agents Against Cross-Modal Attacks
Mar 4, 2026Haoyu Liu, Dingcheng Li, Lukas Rutishauser +1Adversarial Training Multimodal Robustness
UC Berkeley, IEOR & BAIR · Google · Google Deepmind
Feature-level Interaction Explanations in Multimodal Transformers
Mar 4, 2026Yeji Kim, Housam Khalifa Bashier Babiker, Mi-Young Kim +1Transformer Interpretability Feature Attribution
University of Alberta, Edmonton, AB, Canada · University of Waikato, Hamilton, New Zealand · University of Alberta, Camrose, AB, Canada
SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport
Feb 26, 2026Simon Roschmann, Paul Krzakala, Sonia Mazelet +2Cross-Modal Learning Multimodal Embedding
Helmholtz Munich · Technical University of Munich · Munich Center for Machine Learning +3
OmniGAIA: Towards Native Omni-Modal AI Agents
Feb 26, 2026Xiaoxi Li, Wenxiang Jiao, Jiarui Jin +10Multimodal Reasoning AI Agent Benchmarks
1Renmin University of China · 2Xiaohongshu Inc. · 3Peking University +2
LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Models
Feb 15, 2026Shufan Li, Yuchen Zhu, Jiuxiang Gu +6Unified Multimodal Models Multimodal Reasoning
1Adobe · 2UCLA · 3Georgia Tech
UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models
Feb 9, 2026Cheng Yang, Chufan Shi, Bo Shui +7Unified Multimodal Models Image Generation
University of California San Diego · University of Southern California · University of Illinois Urbana-Champaign +1
Quantifying the Gap between Understanding and Generation within Unified Multimodal Models
Feb 2, 2026Chenlong Wang, Yuhang Chen, Zhihan Hu +4Unified Multimodal Models Cross-Modal Alignment
Independent Researcher · University of Maryland · University of Waterloo +1
DeALOG: Decentralized Multi-Agents Log-Mediated Reasoning Framework
Feb 1, 2026Abhijit Chakraborty, Ashish Raj Shekhar, Shiven Agarwal +1Multi-Agent Collaboration Multimodal QA
Arizona State University
Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis
Jan 31, 2026Zicheng Kong, Dehua Ma, Zhenbo Xu +9Reward Modeling RL for Language Models
Beijing University of Posts and Telecommunications, Beijing, P.R. China · ShiFang Technology Inc., Hangzhou, P.R. China · Tsinghua University, Beijing, P.R. China +1
TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs
Jan 30, 2026Baiqi Li, Kangyi Zhao, Ce Zhang +3VLM Evaluation Video Understanding
University of North Carolina at Chapel Hill · University of Pittsburgh · Carnegie Mellon University
SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding
Jan 29, 2026Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum +2Audio-Language Model Evaluation Multimodal Grounding
Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON M5G 1L7, Canada · University of Groningen, Nijenborgh 4, 9747 AG Groningen, Netherlands · York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada
AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
Jan 25, 2026Xilin Jiang, Qiaolin Wang, Junkai Wu +30Audio-Visual Understanding Multimodal Large Language Models
Columbia University · University of Washington · University of Illinois Urbana-Champaign +5
Social Caption: Evaluating Social Understanding in Multimodal Models
Jan 21, 2026Leena Mathur, Bhaavanaa Thumu, Youssouf Kebe +1Multimodal Large Language Models Multimodal Reasoning
School of Computer Science, Carnegie Mellon University
CASHEW: Stabilizing Multimodal Reasoning via Iterative Trajectory Aggregation
Jan 12, 2026Chaoyu Li, Fei Tao, Pooyan FazliTest-Time Scaling for VLMs Test-Time Scaling
Arizona State University · NewsBreak
NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding
Jan 3, 2026Hyeonjeong Ha, Jinjin Ge, Bo Feng +2VLM Evaluation Video Understanding
Apple · University of Illinois Urbana-Champaign
VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing
Dec 12, 2025Emanuel Sánchez Aimar, Gulnaz Zhambulova, Fahad Shahbaz Khan +2Cross-Modal Learning Multimodal Embedding
Linköping University, Sweden
VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
Nov 25, 2025Tianxiang Jiang, Sheng Xia, Yicheng Xu +5VLM Evaluation Multimodal Large Language Models
University of Science and Technology of China · 2Shanghai AI Laboratory · 3Nanjing University +2

Apr 16, 2026, cs.CVJ/K move · Enter open · S save

Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs

Ziyang Luo, Nian Liu, Junwei Han

Northwestern Polytechnical University Xi’an, Shaanxi, China

Multimodal Large Language Models Multimodal Reasoning Multimodal Fusion Multimodal Understanding

Open paper AskPDF arXiv

Omni-modal Large Language Models (Omni-MLLMs) are designed to reason over diverse sensory streams within a unified model. However, we find that multimodal reasoning is not solely determined by sensory evidence, but also by how modality information is organized. Through systematic analysis across diverse reasoning scenarios, we show that different modality organizations exhibit distinct advantages and limitations, and no single topology is universally optimal. Rather than proposing a general-purpose improvement to Omni-MLLM accuracy, we ask a narrower question: can these topology-induced failures be systematically identified and corrected? Motivated by this, we propose Chain of Modality (CoM), a framework that dynamically reorganizes multimodal topology during inference and learns adaptive organization strategies. Experiments across five benchmarks, diverse architectures, and model scales show that CoM reliably recovers topology-sensitive failures, through both training-free planning and lightweight Planner-SFT, while preserving performance on the full benchmark.

Momentum

Related topics

Latest papers 413