Multimodal Reasoning

Follow this topic Ask about this topic

Multimodal reasoning studies how models perceive information across modalities, connect that evidence to knowledge, and use it to answer questions, plan actions, or explain decisions. Common approaches include instruction tuning and chain-of-thought training, reinforcement learning with verifiable rewards, tool-using agents, and structured methods such as graphs or symbolic solvers. Benchmarks such as MathVista and MMMU test visual problem solving, while video, document, scientific, and medical tasks probe reasoning across time and specialized evidence. A central challenge is ensuring that correct answers actually depend on relevant multimodal evidence rather than language shortcuts; grounding, cross-modal integration, and reliable generalization remain difficult.

Latest papers

512 papers47 papers in the last four weeks, down 10% on the four weeks before. 0.3% of all new papers.

Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMs
Apr 7, 2026Hongcheng Liu, Yuhao Wang, Zhe Chen +5Cross-Modal Alignment Multimodal Large Language Models
Shanghai Jiao Tong University
Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
Apr 6, 2026Chaoyou Fu, Haozhi Yuan, Yuhao Dong +16Video Understanding Video-Language Model Evaluation
Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning
Apr 6, 2026Lei Zhang, Junjiao Tian, Zhipeng Fan +9Image Generation T2I Generation
Meta Superintelligence Labs · University of California, San Diego · Worcester Polytechnic Institute +1
The Topology of Multimodal Fusion: Why Current Architectures Fail at Creative Cognition
Apr 6, 2026Xiujiang TanMultimodal Fusion Multimodal Reasoning
Guangzhou Academy of Fine Arts, Guangzhou, China · Guangzhou Academy of Fine Arts
FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning
Apr 4, 2026Zeyu Wang, Jingye Xu, Xiaogang Li +7Scientific Reasoning in Language Models Multimodal Reasoning
Alibaba Group, Beijing, China · Skylenage, Beijing, China
MG $^2$ -RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation
Apr 4, 2026Sijun Dai, Qiang Huang, Xiaoxing You +1Cross-Modal Learning Multimodal RAG
School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Computer Science, Hangzhou Dianzi University
TABQAWORLD: Optimizing Multimodal Reasoning for Multi-Turn Table Question Answering
Apr 3, 2026Tung Sum Thomas Kwok, Xinyu Wang, Xiaofeng Lin +7Table Reasoning Table QA
University of California, Los Angeles · SimpleWay · McGill University +2
Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models
Apr 3, 2026Gengwei Zhang, Jie Peng, Zhen Tan +6RL for VLMs Multimodal Reasoning
University of North Carolina at Chapel Hill · University of Science and Technology of China · Arizona State University +1
LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model
Apr 2, 2026Jiachun Jin, Zetong Zhou, Xiao Yang +4Unified Multimodal Models Multimodal Reasoning
Shanghai Jiao Tong University · Tsinghua University · UCSD
SURE: Synergistic Uncertainty-aware Reasoning for Multimodal Emotion Recognition in Conversations
Apr 2, 2026Yiqiang Cai, Chengyan Wu, Bolei Ma +4Emotion Recognition in Conversations Multimodal Emotion Recognition
Guangdong Provincial Key Laboratory of Quantum Engineering and Quantum Materials · School of Electronic Science and Engineering (School of Microelectronics), South China Normal University · LMU Munich & Munich Center for Machine Learning +2
CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop Reasoning
Apr 2, 2026Junyoung Sung, Seungwoo Lyu, Minjun Kim +3Multimodal Reasoning VLM Reasoning
Dept. of CSE, Korea University · Google DeepMind
MOON3.0: Reasoning-aware Multimodal Representation Learning for E-commerce Product Understanding
Apr 1, 2026Junxian Wu, Chenghan Fu, Zhanheng Nie +6Multimodal Learning Multimodal Large Language Models
Alibaba Group, Hangzhou, China
Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoning
Mar 30, 2026Ziqi Miao, Haonan Jia, Lijun Li +4Multimodal Reasoning Multimodal Large Language Models
BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QA
Mar 30, 2026Taeyun Roh, Suhyeong Park, Dongyoung Lee +4Vision-Language Models Multimodal QA
Korea University · AIGEN Sciences · Myongji University +1
CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs
Mar 30, 2026Yongkang Du, Xiaohan Zou, Minhao Cheng +1Analogical Reasoning Multimodal Reasoning
Pennsylvania State University
MuSEAgent: A Multimodal Reasoning Agent with Stateful Experiences
Mar 29, 2026Shijian Wang, Jiarui Jin, Runhao Fu +11Multimodal Agents Multimodal Reasoning
Southeast University · Monash University · Xiaohongshu Inc. +4
Data Organization Matters in Multimodal Instruction Tuning: A Controlled Study of Capability Trade-offs
Mar 29, 2026Guowei TangMultimodal Instruction Tuning Curriculum Learning
School of Statistics, East China Normal University
Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models
Mar 28, 2026Ji Ma, Wei Suo, Peng Wang +1Multimodal Hallucination Multimodal CoT Reasoning
School of Computer Science and Ningbo Institute, Northwestern Polytechnical University, China. · National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, China.
Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning
Mar 27, 2026Rongjin Li, Zichen Tang, Xianghe Wang +9LLM Evaluation Scientific Reasoning in Language Models
Beijing University of Posts and Telecommunications
Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification
Mar 27, 2026Shuai Lv, Chang Liu, Feng Tang +5Multimodal Large Language Models Multimodal Reasoning
University of Science and Technology of China, Hefei, China · Huawei Foundation Model Department, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China +1
Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR
Mar 27, 2026Jinda Lu, Junkang Wu, Jinghan Li +6RL for VLMs Multimodal Reasoning
University of Science and Technology of China · Pecking University · Institute of Software, Chinese Academy of Sciences
R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoning
Mar 26, 2026Zirui Zhang, Haoyu Dong, Kexin Pei +1Multimodal Reasoning Cross-Modal Learning
Rutgers University · Columbia University · University of Chicago
AD-Reasoning: Multimodal Guideline-Guided Reasoning for Alzheimer's Disease Diagnosis
Mar 25, 2026Qiuhui Chen, Yushan Deng, Xuancheng Yao +1Alzheimer's Disease Classification Medical Diagnosis
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning
Mar 25, 2026Kun-Yang Yu, Zhi Zhou, Shi-Yu Tian +6Multimodal Understanding Multimodal Large Language Models
State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China
TED: Training-Free Experience Distillation for Multimodal Reasoning
Mar 25, 2026Shuozhi Yuan, Jinqing Wang, Zihao Liu +5Context Distillation Knowledge Distillation
China Telecom Digital Intelligence Technology Co.,Ltd., Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
From Prediction to Diagnosis: Reasoning-Aware AI for Photovoltaic Defect Inspection
Mar 24, 2026Dev Mistry, Feng Qiu, Bo Chen +4Fault Detection Industrial Inspection
Department of Electrical and Computer Engineering, Illinois Institute of Technology, Chicago, IL 60616, USA · Argonne National Laboratory, Lemont, IL 60439, USA · Smart Grid Emerging Technology Department, Commonwealth Edison, Chicago, IL 60618, USA +2
ENC-Bench: A Benchmark for Evaluating Multimodal Large Language Models in Electronic Navigational Chart Understanding
Mar 24, 2026Ao Cheng, Xingming Li, Xuanyu Ji +5Multimodal Spatial Reasoning Maritime Navigation
National University of Defense Technology · Intelligent Game and Decision Lab · The Chinese University of Hong Kong, Shenzhen
Detecting Corporate AI-Washing via Cross-Modal Semantic Inconsistency Learning
Mar 24, 2026Zhanjie Wen, Jingqiao GuoMultimodal Reasoning
School of Economics and Trade, Guangdong University of Finance, Guangzhou 510521, China · Department of Computer Science, Faculty of Science, Hong Kong Baptist University, Hong Kong 999077, China
Think 360°: Evaluating the Width-centric Reasoning Capability of MLLMs Beyond Depth
Mar 24, 2026Mingrui Chen, Hexiong Yang, Haogeng Liu +2Multimodal Reasoning Multimodal Model Evaluation
School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR&MAIS, Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy +1
More Than Sum of Its Parts: Deciphering Intent Shifts in Multimodal Hate Speech Detection
Mar 22, 2026Runze Sun, Yu Zheng, Zexuan Xiong +4Hate Speech Detection Multimodal Reasoning
Department of Automation, Tsinghua University

Apr 7, 2026J/K move · Enter open · S save

Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMs

Hongcheng Liu, Yuhao Wang, Zhe Chen, Pingjie Wang+4

Shanghai Jiao Tong University

Cross-Modal Alignment Multimodal Large Language Models Coreference Resolution Multimodal Reasoning

Open paper AskPDF arXiv

Omni Large Language Models (Omni-LLMs) have demonstrated impressive capabilities in holistic multi-modal perception, yet they consistently falter in complex scenarios requiring synergistic omni-modal reasoning. Beyond understanding global multimodal context, effective reasoning also hinges on fine-grained cross-modal alignment, especially identifying shared referents across modalities, yet this aspect has been largely overlooked. To bridge this gap, we formalize the challenge as a cross-modal coreference problem, where a model must localize a referent in a source modality and re-identify it in a target modality. Building on this paradigm, we introduce CrossOmni, a dataset comprising nine tasks equipped with human-designed reasoning rationales to evaluate and enhance this capability. Experiments on 13 Omni-LLMs reveal systematic weaknesses in cross-modal coreference, which we attribute to the absence of coreference-aware thinking patterns. To address this, we enhance cross-modal alignment via two strategies: a training-free In-Context Learning method and a training-based SFT+GRPO framework designed to induce such thinking patterns. Both approaches yield substantial performance gains and generalize effectively to collaborative reasoning tasks. Overall, our findings highlight cross-modal coreference as a crucial missing piece for advancing robust omni-modal reasoning.