Multimodal Reasoning

Follow this topic Ask about this topic

Multimodal reasoning studies how models perceive information across modalities, connect that evidence to knowledge, and use it to answer questions, plan actions, or explain decisions. Common approaches include instruction tuning and chain-of-thought training, reinforcement learning with verifiable rewards, tool-using agents, and structured methods such as graphs or symbolic solvers. Benchmarks such as MathVista and MMMU test visual problem solving, while video, document, scientific, and medical tasks probe reasoning across time and specialized evidence. A central challenge is ensuring that correct answers actually depend on relevant multimodal evidence rather than language shortcuts; grounding, cross-modal integration, and reliable generalization remain difficult.

Latest papers

512 papers47 papers in the last four weeks, down 10% on the four weeks before. 0.3% of all new papers.

Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems
Apr 16, 2026Nishanth Madhusudhan, Vikas Yadav, Alexandre LacosteMultimodal QA Multimodal Reasoning
ServiceNow Research
Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs
Apr 16, 2026Ziyang Luo, Nian Liu, Junwei HanMultimodal Large Language Models Multimodal Reasoning
Northwestern Polytechnical University Xi’an, Shaanxi, China
Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarios
Apr 15, 2026Xiaomin Li, Tala Wang, Zichen Zhong +7VLM Evaluation Multimodal Reasoning
Dalian University of Technology · WeChat, Tencent Inc. · Tsinghua University
MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments
Apr 15, 2026Han Wang, David Wan, Hyunji Lee +6AI Agent Benchmarks Multimodal Search Agents
UNC Chapel Hill · Virginia Tech · University of Texas at Austin
Medical thinking with multiple images
Apr 14, 2026Zonghai Yao, Benlu Wang, Yifan Zhang +8Medical VQA Multimodal Reasoning
Manning College of Information and Computer Sciences, UMass Amherst · Center for Healthcare Organization and Implementation Research, VA Bedford Health Care · Department of Computer Science, Yale University +2
Towards Long-horizon Agentic Multimodal Search
Apr 14, 2026Yifan Du, Zikang Liu, Jinbiao Peng +5Multimodal Search Agents Long-Horizon LLM Agents
Gaoling School of Artificial Intelligence, Renmin University of China. · City University of Hong Kong.
MISID: A Multimodal Multi-turn Dataset for Complex Intent Recognition in Strategic Deception Games
Apr 14, 2026Shufang Lin, Muyang Chen, Xiabing Zhou +3Multimodal Intent Recognition Multimodal Large Language Models
The Chinese University of Hong Kong, Shenzhen Shenzhen, GuangDong, China · Capital Normal University Beijing, China
MODIX: A Training-Free Multimodal Information-Driven Positional Index Scaling for Vision-Language Models
Apr 14, 2026Ruoxiang Huang, Zhen YuanVision-Language Models VLM Adaptation
Peking University, China
Modality-Native Routing in Agent-to-Agent Networks: A Multimodal A2A Protocol Extension
Apr 14, 2026Vasundra SrinivasanMulti-Agent Communication Multimodal Model Routing
AI Architect, Author—Data Engineering for Multimodal AI (O’Reilly), Stanford School of Engineering
Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Games
Apr 13, 2026Keyang Zhong, Junlin Xie, Hefeng Wu +2VLM Reasoning Multimodal Reasoning
Sun Yat-sen University · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen
CFMS: A Coarse-to-Fine Multimodal Synthesis Framework for Enhanced Tabular Reasoning
Apr 13, 2026Qixian Huang, Hongqiang Lin, Tong Fu +5Table QA Multimodal Reasoning
When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalities
Apr 12, 2026Sarmistha Das, Shreyas Guha, Suvrayan Bandyopadhyay +3Retrieval-Augmented Reasoning Figurative Language Understanding
Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, India · School of Information Technology, King Mongkut’s Institute of Technology Ladkrabang, Bangkok, Thailand
SVSR: A Self-Verification and Self-Rectification Paradigm for Multimodal Reasoning
Apr 11, 2026Zhe Qian, Nianbing Su, Zhonghua Wang +6Multimodal Reasoning Vision-Language Models
South China Agricultural University · Monash University · University of Glasgow +5
Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Models
Apr 11, 2026Zhe Qian, Yanbiao Ma, Zhuohan Ouyang +7VLM Hallucination Mitigation Multimodal Reasoning
South China Agricultural University · Gaoling School of Artificial Intelligence, Renmin University of China · South China Normal University +4
GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensing
Apr 10, 2026Aoran Xiao, Shihao Cheng, Yonghao Xu +3Remote Sensing Benchmarks Remote Sensing VQA
RIKEN AIP · Wuhan University · Linköping University +2
Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Models
Apr 9, 2026Sumra Khan, Sagar Chhabriya, Aizan Zafar +5VLM Hallucination Mitigation Medical VLMs
Department of Computer Science, Salim Habib University, Karachi, Pakistan · Computer Science, Institute of Business Administration Sukkur, Pakistan · Center for Research in Computer Vision, University of Central Florida, USA +2
Unified Multimodal Uncertain Inference
Apr 9, 2026Dengjia Zhang, Alexander Martin, William Jurayj +3Probability Calibration Multimodal Reasoning
Johns Hopkins University · Human Language Technology Center of Excellence
Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts
Apr 9, 2026Haolei Xu, Haiwen Hong, Hongxing Li +7VLM Reasoning Expert Routing
Zhejiang University · Alibaba Group
OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks
Apr 9, 2026Wenbo Hu, Xin Chen, Yan Gao-Tian +3Multimodal Large Language Models Multimodal Reasoning
University of California, Los Angeles (UCLA)
OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering
Apr 9, 2026Yiduo Jia, Muzhi Zhu, Hao Zhong +7Audio-Visual Understanding Multimodal Reasoning
Zhejiang University · Xiaomi Inc.
Multimodal Reasoning with LLM for Encrypted Traffic Interpretation: A Benchmark
Apr 9, 2026Longgang Zhang, Xiaowei Fu, Fuxiang Huang +1Encrypted Network Traffic Analysis Encrypted Traffic Classification
School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China. · School of Data Science, Lingnan University, Hong Kong, China.
Multimodal Latent Reasoning via Predictive Embeddings
Apr 9, 2026Ashutosh Adhikari, Mirella LapataVLM Reasoning Multimodal Reasoning
School of Informatics University of Edimburgh
M-ArtAgent: Evidence-Based Multimodal Agent for Implicit Art Influence Discovery
Apr 8, 2026Hanyi Liu, Zhonghao Jiu, Minghao Wang +2Multimodal Agents Multimodal Reasoning
China Electronics Technology Group Co., Ltd., Beijing 100043, China · School of Information Science and Engineering, Southeast University, Nanjing 211189, China · Department of Chemical and Biological Engineering, Hong Kong University of Science and Technology, Hong Kong SAR, China +2
What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning
Apr 8, 2026Songze Li, Xiaoke Guo, Tianqi Liu +5GUI Grounding Multimodal Reasoning
Zhejiang University · ZJU-Ant Group Joint Lab of Knowledge Graph
Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization
Apr 8, 2026Wenhao Yang, Yu Xia, Jinlong Huang +10Multimodal Reasoning Multimodal Agents
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · AI Business, Alibaba Group +5
LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models
Apr 8, 2026Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu +5Visual Spatial Reasoning Multimodal Large Language Models
National Key Laboratory for Novel Software Technology, Nanjing University, China
RASR: Retrieval-Augmented Semantic Reasoning for Fake News Video Detection
Apr 8, 2026Hui Li, Peien Ding, Jun Li +5Multimodal RAG Fake News Detection
School of Informatics, Xiamen University, Xiamen, China · School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin, China · School of Computer and Big Data, Minjiang University, Fuzhou, China +2
MTA-Agent: An Open Recipe for Multimodal Deep Search Agents
Apr 7, 2026Xiangyu Peng, Can Qin, An Yan +4Multimodal Search Agents Multimodal QA
Salesforce AI Research
MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control
Apr 7, 2026Yuchi Wang, Dingkang Yang, Haiyang Yu +5Multimodal Embedding Multimodal Reasoning
MMLab, The Chinese University of Hong Kong · ByteDance Inc. · Fudan University
EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agents
Apr 7, 2026Xuan Dong, Huanyang Zheng, Tianhao Niu +7AI Agent Benchmarks AI Agents for Scientific Discovery
Harbin Institute of Technology

Apr 16, 2026J/K move · Enter open · S save

Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems

Nishanth Madhusudhan, Vikas Yadav, Alexandre Lacoste

ServiceNow Research

Multimodal QA Multimodal Reasoning Multimodal Agents Multimodal Model Evaluation Multimodal Understanding

Open paper AskPDF arXiv

Effective abstention (EA), recognizing evidence insufficiency and refraining from answering, is critical for reliable multimodal systems. Yet existing evaluation paradigms for vision-language models (VLMs) and multi-agent systems (MAS) assume answerability, pushing models to always respond. Abstention has been studied in text-only settings but remains underexplored multimodally; current benchmarks either ignore unanswerability or rely on coarse methods that miss realistic failure modes. We introduce MM-AQA, a benchmark that constructs unanswerable instances from answerable ones via transformations along two axes: visual modality dependency and evidence sufficiency. Evaluating three frontier VLMs spanning closed and open-source models and two MAS architectures across 2079 samples, we find: (1) under standard prompting, VLMs rarely abstain; even simple confidence baselines outperform this setup, (2) MAS improves abstention but introduces an accuracy-abstention trade-off, (3) sequential designs match or exceed iterative variants, suggesting the bottleneck is miscalibration rather than reasoning depth, and (4) models abstain when image or text evidence is absent, but attempt reconciliation with degraded or contradictory evidence. Effective multimodal abstention requires abstention-aware training rather than better prompting or more agents.