Multimodal Understanding

Follow this topic Ask about this topic

The field studies how AI can combine evidence from different kinds of input to answer questions, recognize events, retrieve information, and reason about scenes over time. Common approaches include vision-language models and multimodal large language models, trained with contrastive alignment, cross-attention, instruction tuning, and reinforcement learning; some systems add modality-specific experts, retrieval, or tools. Benchmarks test image and video question answering, document and chart understanding, audio-visual reasoning, grounding, and performance across languages and domains. A central challenge is ensuring that answers genuinely depend on relevant evidence from each modality, while handling long inputs efficiently and making capabilities measurable with reliable tests.

Latest papers

479 papers15 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis
Jan 31, 2026Zicheng Kong, Dehua Ma, Zhenbo Xu +9Reward Modeling RL for Language Models
Beijing University of Posts and Telecommunications, Beijing, P.R. China · ShiFang Technology Inc., Hangzhou, P.R. China · Tsinghua University, Beijing, P.R. China +1
TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs
Jan 30, 2026Baiqi Li, Kangyi Zhao, Ce Zhang +3VLM Evaluation Video Understanding
University of North Carolina at Chapel Hill · University of Pittsburgh · Carnegie Mellon University
SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding
Jan 29, 2026Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum +2Multimodal Understanding Multimodal Grounding
Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON M5G 1L7, Canada · University of Groningen, Nijenborgh 4, 9747 AG Groningen, Netherlands · York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada
MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models
Jan 29, 2026Sangyun Chung, Se Yeon Kim, Youngchae Chee +1Multimodal Robustness Hallucination in Language Models
Integrated Vision Language Lab, KAIST, South Korea
Uni-RS: A Spatially Faithful Unified Understanding and Generation Model for Remote Sensing
Jan 25, 2026Weiyu Zhang, Yuan Hu, Yong Li +1Unified Multimodal Models Multimodal Generation
Institute of Remote Sensing and Geographic Information System, School of Earth and Space Sciences, Peking University, Beijing, 100871, China · Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong
AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
Jan 25, 2026Xilin Jiang, Qiaolin Wang, Junkai Wu +30Audio-Visual Understanding Multimodal Large Language Models
Columbia University · University of Washington · University of Illinois Urbana-Champaign +5
Social Caption: Evaluating Social Understanding in Multimodal Models
Jan 21, 2026Leena Mathur, Bhaavanaa Thumu, Youssouf Kebe +1Multimodal Large Language Models Multimodal Reasoning
School of Computer Science, Carnegie Mellon University
CASHEW: Stabilizing Multimodal Reasoning via Iterative Trajectory Aggregation
Jan 12, 2026Chaoyu Li, Fei Tao, Pooyan FazliTest-Time Scaling for VLMs Test-Time Scaling
Arizona State University · NewsBreak
NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding
Jan 3, 2026Hyeonjeong Ha, Jinjin Ge, Bo Feng +2VLM Evaluation Video Understanding
Apple · University of Illinois Urbana-Champaign
VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing
Dec 12, 2025Emanuel Sánchez Aimar, Gulnaz Zhambulova, Fahad Shahbaz Khan +2Multimodal Understanding Cross-Modal Learning
Linköping University, Sweden
VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
Nov 25, 2025Tianxiang Jiang, Sheng Xia, Yicheng Xu +5VLM Evaluation Multimodal Large Language Models
University of Science and Technology of China · 2Shanghai AI Laboratory · 3Nanjing University +2
SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards
Nov 10, 2025Hunar Batra, Haoqin Tu, Hardy Chen +3Multimodal Understanding Scene Graph Generation
University of Oxford · University of California, Santa Cruz
M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset
Oct 27, 2025Jiahui Geng, Jonathan Tonglet, Iryna GurevychAutomated Fact-Checking Multimodal Understanding
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany · Department of Electrical Engineering, KU Leuven, Belgium +2
From Charts to Code: A Hierarchical Benchmark for Multimodal Models
Oct 20, 2025Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu +8Data Visualization Chart-to-Code Generation
CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University
Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models
Sep 26, 2025Jiawei Liang, Jianjie Huang, Ruoyu Chen +4Feature Attribution Multimodal Large Language Models
Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China · Zhongguan-cun Academy, Beijing 100094, China · University of Chinese Academy of Sciences, Beijing 100049, China +2
Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation
Sep 23, 2025Shufan Li, Jiuxiang Gu, Kangning Liu +4Masked Diffusion Models Unified Multimodal Models
Adobe · UCLA
Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy
Sep 22, 2025Geewook Kim, Minjoon SeoVLM Evaluation Audio-Visual Understanding
NAVER Cloud AI, South Korea · KAIST AI, South Korea
Reconstruction Alignment Improves Unified Multimodal Models
Sep 8, 2025Ji Xie, Trevor Darrell, Luke Zettlemoyer +1Unified Multimodal Models Cross-Modal Alignment
1UC Berkeley · University of Washington · 3Duke University
VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark
Aug 19, 2025Vy Tuong Dang, An Vo, Emilio Villa-Cueva +4VLM Evaluation Multimodal Reasoning
KAIST · MBZUAI
Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping
Aug 17, 2025Xuhui Zhan, Tyler DerrCross-Modal Learning Cross-Modal Representation Learning
Data Science Institute, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States. · Computer Science Department, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States.
Position: Reasoning After Perception Means Reasoning Without Vision
Jul 21, 2025Hongcheng Gao, Zihao Huang, Jingyi Tang +12Multimodal Understanding Visual Reasoning
Tsinghua University · BIT · Peking University +6
Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
Jun 8, 2025Weiwen Xu, Hou Pong Chan, Long Li +15Medical Report Generation Multimodal Pretraining
LASA Team · DAMO Academy, Alibaba Group
How Well Do Vision-Language Models Understand Movies?
Jun 6, 2025Emmanouil Zaranis, António Farinhas, Saul Santos +28Video-Language Model Evaluation Multimodal Understanding
Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · ILLC, University of Amsterdam +12
Mitigating Task Objective Conflicts for Continual Learning in Unified Multimodal Models
Jun 4, 2025Jiaxing Zhang, Hao TangTask Interference Unified Multimodal Models
Sichuan University, Chengdu, China · Peking University, Beijing, China
LaViDa: A Large Diffusion Language Model for Multimodal Understanding
May 22, 2025Shufan Li, Konstantinos Kallidromitis, Hritik Bansal +7Multimodal Understanding Vision-Language Models
1UCLA · 2Panasonic AI Research · 4Salesforce Research +1
HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation
May 16, 2025Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie +6Algorithmic Fairness Multimodal Large Language Models
Vector Institute for Artificial Intelligence, Toronto, Canada · University of Central Florida, Orlando, USA
ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models
Feb 13, 2025Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma +31Visually Grounded Reasoning VLM Reasoning
University of Cambridge
MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis
Oct 19, 2024Junda Wang, Zonghai Yao, Yujan Ting +5Healthcare Medical Image Analysis
United Imaging Intelligence, MA, USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, MA, USA · Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA +1
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
Jul 16, 2024Haodong Duan, Xinyu Fang, Junming Yang +41VLM Evaluation LLM Evaluation

Jan 31, 2026J/K move · Enter open · S save

Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis

Zicheng Kong, Dehua Ma, Zhenbo Xu, Alven Yang+8

Beijing University of Posts and Telecommunications, Beijing, P.R. China · ShiFang Technology Inc., Hangzhou, P.R. China · Tsinghua University, Beijing, P.R. China+1

Reward Modeling RL for Language Models Synthetic Data Generation Multimodal Reward Modeling Multimodal Understanding

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) struggle with alignment due to the limitations of existing reward models (RMs), which are predominantly vision-centric, dependent on costly human labels, and provide opaque scalar scores that fail to capture nuanced reasoning, leading to brittle alignment. We present Omni-RRM, an Omni-modal Rubric-grounded Reward Model that generates multi-dimensional reward signals across text, image, video, and audio. To overcome the high cost and inherent inconsistency of human-centric evaluation in multi-dimensional reasoning, we introduce Omni-Preference, a high-quality dataset constructed via automatic rubric-grounded preference synthesis. In this pipeline, teacher models reconcile raw preferences into explicit justifications, ensuring that the synthesized supervision is both high-fidelity and interpretable. Omni-RRM is trained using a progressive SFT + GRPO regimen, specifically optimized to sharpen reward discrimination on low-margin, hard preference pairs. It achieves state-of-the-art accuracy on video (80.2% on ShareGPT-Video) and audio benchmarks (66.8% on Audio-HH-RLHF and 65.0% on TA2T), yielding a five-benchmark Overall accuracy of 70.4% and a +17.0% relative gain over its backbone. Furthermore, Omni-RRM effectively guides Best-of-

N

selection and exhibits robust transfer to text-only alignment. All resources, including the dataset, training and inference code, and model checkpoints are available at https://tmfk418.github.io/Omni-RRM.