Multimodal Reasoning

Follow this topic Ask about this topic

Multimodal reasoning studies how models perceive information across modalities, connect that evidence to knowledge, and use it to answer questions, plan actions, or explain decisions. Common approaches include instruction tuning and chain-of-thought training, reinforcement learning with verifiable rewards, tool-using agents, and structured methods such as graphs or symbolic solvers. Benchmarks such as MathVista and MMMU test visual problem solving, while video, document, scientific, and medical tasks probe reasoning across time and specialized evidence. A central challenge is ensuring that correct answers actually depend on relevant multimodal evidence rather than language shortcuts; grounding, cross-modal integration, and reliable generalization remain difficult.

Latest papers

512 papers47 papers in the last four weeks, down 10% on the four weeks before. 0.3% of all new papers.

LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations
May 1, 2026Huangbiao Xu, Huanqi Wu, Xiao Ke +1Missing-Modality Learning Multimodal Reasoning
Fujian Provincial Key Laboratory of Networking Computing and Intelligent Information Processing, College of Computer and Data Science, Fuzhou University, Fuzhou 350108, China · Engineering Research Center of Big Data Intelligence, Ministry of Education, Fuzhou 350108, China · Wangxuan Institute of Computer Technology, Peking University, Beijing 100871, China
Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL
Apr 30, 2026Sudong Wang, Weiquan Huang, Xiaomin Yu +9Multimodal Large Language Models Multimodal Reasoning
Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University · Tsinghua University +3
Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation
Apr 30, 2026Jing Zhang, Wentao Jiang, Tao Huang +8Medical Image Analysis Multimodal Reasoning
School of Computer Science, Wuhan University, China · The Central Hospital of Wuhan, China · Taizhou Hospital of Zhejiang Province, China +1
MM-StanceDet: Retrieval-Augmented Multi-modal Multi-agent Stance Detection
Apr 30, 2026Weihai Lu, Zhejun Zhao, Yanshu Li +1Multimodal Reasoning Multimodal Classification
Peking University · Baidu Inc · Brown University +1
GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
Apr 29, 2026GLM-V Team, :, Wenyi Hong +95Multimodal Large Language Models Multimodal Reasoning
Omni-o3: Deep Nested Omnimodal Deduction for Deliberative Audio-Visual Reasoning
Apr 27, 2026Zhicheng Zhang, Wentao Gu, Weicheng Wang +5Audio-Visual Understanding Tree Search
Nankai University · Kuaishou Technology · Pengcheng Laboratory +1
DeepTaxon: An Interpretable Retrieval-Augmented Multimodal Framework for Unified Species Identification and Discovery
Apr 27, 2026Jiawei Wang, Ming Lei, Yaning Yang +8Retrieval-Augmented Classification AI-Assisted Scientific Research
National University of Singapore · Hunan University · Xiangtan University +4
ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction
Apr 26, 2026Zichun Guo, Yuling Shi, Wenhao Zeng +6Image Reconstruction Multimodal Large Language Models
Xidian University · Shanghai Jiao Tong University · Alibaba Qwen +1
Multimodal QUD: Inquisitive Questions from Scientific Figures
Apr 26, 2026Yating Wu, William Rudman, Venkata S Govindarajan +2Multimodal Grounding Scientific QA
The University of Texas at Austin · Ithaca College · UC Berkeley, BespokeLabs.ai
StoryTR: Narrative-Centric Video Temporal Retrieval with Theory of Mind Reasoning
Apr 25, 2026Xuanyue Zhong, Yuqiang Xie, Guanqun Bi +2Temporal Video Grounding Video Understanding
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong · SkyReels, Skywork AI · Independent Researcher
Context Unrolling in Omni Models
Apr 23, 2026Ceyuan Yang, Zhijie Lin, Yang Zhao +16Cross-Modal Learning Unified Multimodal Models
S1-VL: Scientific Multimodal Reasoning Model with Thinking-with-Images
Apr 23, 2026Qingxiao Li, Lifeng Xu, QingLi Wang +4Tool Use in VLMs Multimodal Reasoning
ScienceOne AI
Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning
Apr 22, 2026Dahun Kim, Ganesh Satish Mallya, Anelia AngelovaRemote Sensing Image Understanding Multimodal Large Language Models
Google DeepMind
OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model
Apr 22, 2026Qiguang Chen, Chengyu Luan, Jiajun Wu +7VLM Evaluation Visual Reasoning
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Central South University · Fudan University +3
V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimization
Apr 22, 2026Yubo Jiang, Yitong An, Xin Yang +7Multimodal Reasoning Reinforcement Learning with Verifiable Rewards
School of Astronautics, Beihang University, Beijing 102206, China · Longcat Interaction Team, Meituan, Beijing 100102, China · Tianmushan Laboratory, Beihang University, Hangzhou 311115, China
ONOTE: Benchmarking Omnimodal Notation Processing for Expert-level Music Intelligence
Apr 22, 2026Menghe Ma, Siqing Wei, Yuecheng Xing +5Optical Music Recognition Multimodal Reasoning
Beijing University of Posts and Telecommunications Beijing, China · China Conservatory of Music Beijing, China · Nanyang Technological University Singapore, Singapore
The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning Within the Monolithic Paradigm
Apr 22, 2026Karan GoyalVLM Evaluation Multimodal Reasoning
IIIT Delhi, India
HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimization
Apr 22, 2026Tao Cheng, Shi-Zhe Chen, Hao Zhang +3Multimodal Large Language Models Multimodal Reasoning
Tencent PCG, China · Tencent CSIG, China
Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks
Apr 21, 2026Jing Jin, Hao Liu, Yan Bai +9Reasoning Evaluation Multimodal Reasoning
1Tsinghua University · 3Central South University · 2Meituan Inc +3
How Far Are Video Models from True Multimodal Reasoning?
Apr 21, 2026Xiaotian Zhang, Jianhui Wei, Yuan Wang +9Multimodal Reasoning Video Reasoning
Zhejiang University · ByteDance
A Multi-Agent Framework with Structured Reasoning and Reflective Refinement for Multimodal Empathetic Response Generation
Apr 21, 2026Liping Wang, Cheng Ye, Weidong Chen +3LLM Self-Refinement Multimodal Reasoning
School of Information Science and Technology, USTC Hefei, China
Prioritizing the Best: Incentivizing Reliable Multimodal Reasoning by Rewarding Beyond Answer Correctness
Apr 20, 2026Mengzhao Jia, Zhihan Zhang, Meng JiangMultimodal Reasoning Multimodal Reward Modeling
University of Notre Dame
DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning
Apr 20, 2026Abrar Majeedi, Zhiyuan Ruan, Ziyi Zhao +3Cross-Modal Learning Visual Question Answering
University of Wisconsin-Madison · Amazon · Northeastern University
Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs
Apr 20, 2026Samuel G. Balter, Ethan Jerzak, Connor T. JerzakMultimodal Large Language Models Multimodal Reasoning
University of Texas at Austin · National University of Singapore (NUS)
Retrieval-Augmented Multimodal Model for Fake News Detection
Apr 20, 2026Yiheng Li, Weihai Lu, Hanyi Yu +1Fake News Detection Text Classification
University of International Business and Economics · Beijing, China · Peking University +4
AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers
Apr 17, 2026Edson Araujo, Saurabhchand Bhati, M. Jehanzeb Mirza +5Cross-Modal Learning Audio-Visual Understanding
University of Tübingen, Germany · MIT, Cambridge MA, USA · IBM Research, USA +2
Real-Time Visual Attribution Streaming in Thinking Model
Apr 17, 2026Seil Kang, Woojung Han, Junhyeok Kim +3VLM Interpretability Multimodal Reasoning
YYonsei University. · AAmazon.
Watching Movies Like a Human: Egocentric Emotion Understanding for Embodied Companions
Apr 17, 2026Ze Dong, Hao Shi, Zejia Gao +3Multimodal Robustness Egocentric Video Understanding
Nanyang Technological University, Singapore · Zhejiang University, China · Ant Group, China +1
Towards Robust Endogenous Reasoning: Unifying Drift Adaptation in Non-Stationary Tuning
Apr 17, 2026Xiaoyu Yang, En Yu, Wei Duan +1Multimodal Robustness Multimodal Large Language Models
Australian Artificial Intelligence Institute (AAII), Faculty of Engineering and Information Technology, University of Technology Sydney, Australia
PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation
Apr 17, 2026Shuyan Ke, Yifan Mei, Changli Wu +4Image Segmentation Remote Sensing Image Segmentation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Shanghai Innovation Institute

May 1, 2026J/K move · Enter open · S save

LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations

Huangbiao Xu, Huanqi Wu, Xiao Ke, Yuxin Peng

Fujian Provincial Key Laboratory of Networking Computing and Intelligent Information Processing, College of Computer and Data Science, Fuzhou University, Fuzhou 350108, China · Engineering Research Center of Big Data Intelligence, Ministry of Education, Fuzhou 350108, China · Wangxuan Institute of Computer Technology, Peking University, Beijing 100871, China

Missing-Modality Learning Multimodal Reasoning Multimodal Fusion Action Quality Assessment Multimodal Learning Multimodal Understanding

Open paper AskPDF arXiv

Real-world multimodal learning is often hindered by missing modalities. While Incomplete Multimodal Learning (IML) has gained traction, existing methods typically rely on the unrealistic assumption of full-modal availability during training to provide reconstruction supervision or cross-modal priors. This paper tackles the more challenging setting of IML under training-time incomplete observations, which precludes reliance on a ``God's eye view'' of complete data. We propose LIMSSR (LLM-Driven Incomplete Multimodal Sequence-to-Score Reasoning), a framework that reformulates this challenge as a conditional sequence reasoning task. LIMSSR leverages the semantic reasoning capabilities of Large Language Models via Prompt-Guided Context-Aware Modality Imputation and Multidimensional Representation Fusion to infer latent semantics from available contexts without direct reconstruction. To mitigate hallucinations, we introduce a Mask-Aware Dual-Path Aggregation to dynamically calibrate inference uncertainty. Extensive experiments on three Action Quality Assessment datasets demonstrate that LIMSSR significantly outperforms state-of-the-art baselines without relying on complete training data, establishing a new paradigm for data-efficient multimodal learning. Code is available at https://github.com/XuHuangbiao/LIMSSR.