Multimodal Understanding

Ask about this topic

Momentum

10 papers in the last four weeks, up 233% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 413

All topics
CardsList

  1. Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs

    Apr 16, 2026Ziyang Luo, Nian Liu, Junwei HanMultimodal Large Language ModelsMultimodal Reasoning

    Northwestern Polytechnical University Xi’an, Shaanxi, China

  2. Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs

    Apr 13, 2026Kaizhen Tan, Yang Feng, Heqing Du +3VLM EvaluationConfidence Estimation in Language Models

    Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA

  3. Unified Multimodal Uncertain Inference

    Apr 9, 2026Dengjia Zhang, Alexander Martin, William Jurayj +3Probability CalibrationMultimodal Reasoning

    Johns Hopkins University · Human Language Technology Center of Excellence

  4. MG2^2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

    Apr 4, 2026Sijun Dai, Qiang Huang, Xiaoxing You +1Cross-Modal LearningMultimodal RAG

    School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Computer Science, Hangzhou Dianzi University

  5. Internalized Reasoning for Long-Context Visual Document Understanding

    Mar 31, 2026Austin VeselkaLong-Context ModelingVLM Reasoning

    LightOn

  6. AMIGO: Agentic Multi-Image Grounding Oracle Benchmark

    Mar 30, 2026Min Wang, Ata MahjoubfarVLM EvaluationVisual Question Answering

    Target Corporation

  7. BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QA

    Mar 30, 2026Taeyun Roh, Suhyeong Park, Dongyoung Lee +4Vision-Language ModelsMultimodal QA

    Korea University · AIGEN Sciences · KAIST +1

  8. UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy

    Mar 25, 2026Yicheng Xu, Jiangning Zhang, Zhucun Xue +5Multimodal ICLUnified Multimodal Models

    1Zhejiang University, China. · 2Shanghai Jiaotong University, China. · 3National University of Singapore, Singapore. +1

  9. UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

    Mar 23, 2026Ziyi Wang, Xinshun Wang, Shuang Chen +2Cross-Modal LearningUnified Multimodal Models

    State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China · Donghua University, China · School of Automation Science and Engineering, South China University of Technology

  10. HumanOmni-Speaker: Identifying Who said What and When

    Mar 23, 2026Detao Bai, Zhiheng Ma, Xihan WeiAudio-Visual UnderstandingVisual Speaker Recognition

    Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology · Guangdong Provincial Key Laboratory of Computility Microelectronics +1

  11. ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models

    Mar 19, 2026Thomas De Min, Subhankar Roy, Stéphane Lathuilière +2Multimodal Large Language ModelsAI Agent Benchmarks

    University of Trento · University of Bergamo · Inria, Univ. Grenoble Alpes, CNRS, LJK +1

  12. HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering

    Mar 19, 2026Dan Ben-Ami, Gabriele Serussi, Kobi Cohen +1Multimodal Large Language ModelsMultimodal QA

    INSIGHT Lab, Ben-Gurion University of the Negev, Israel · Ben-Gurion University of the Negev, Israel

  13. Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion

    Mar 6, 2026Lijiang Li, Zuwei Long, Yunhang Shen +6Masked Diffusion ModelsUnified Multimodal Models

    Nanjing University · Tencent Youtu Lab · CASIA

  14. Dual-Modality Multi-Stage Adversarial Safety Training: Robustifying Multimodal Web Agents Against Cross-Modal Attacks

    Mar 4, 2026Haoyu Liu, Dingcheng Li, Lukas Rutishauser +1Adversarial TrainingMultimodal Robustness

    UC Berkeley, IEOR & BAIR · Google · Google Deepmind

  15. Feature-level Interaction Explanations in Multimodal Transformers

    Mar 4, 2026Yeji Kim, Housam Khalifa Bashier Babiker, Mi-Young Kim +1Transformer InterpretabilityFeature Attribution

    University of Alberta, Edmonton, AB, Canada · University of Waikato, Hamilton, New Zealand · University of Alberta, Camrose, AB, Canada

  16. SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport

    Feb 26, 2026Simon Roschmann, Paul Krzakala, Sonia Mazelet +2Cross-Modal LearningMultimodal Embedding

    Helmholtz Munich · Technical University of Munich · Munich Center for Machine Learning +3

  17. OmniGAIA: Towards Native Omni-Modal AI Agents

    Feb 26, 2026Xiaoxi Li, Wenxiang Jiao, Jiarui Jin +10Multimodal ReasoningAI Agent Benchmarks

    1Renmin University of China · 2Xiaohongshu Inc. · 3Peking University +2

  18. LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Models

    Feb 15, 2026Shufan Li, Yuchen Zhu, Jiuxiang Gu +6Unified Multimodal ModelsMultimodal Reasoning

    1Adobe · 2UCLA · 3Georgia Tech

  19. UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models

    Feb 9, 2026Cheng Yang, Chufan Shi, Bo Shui +7Unified Multimodal ModelsImage Generation

    University of California San Diego · University of Southern California · University of Illinois Urbana-Champaign +1

  20. Quantifying the Gap between Understanding and Generation within Unified Multimodal Models

    Feb 2, 2026Chenlong Wang, Yuhang Chen, Zhihan Hu +4Unified Multimodal ModelsCross-Modal Alignment

    Independent Researcher · University of Maryland · University of Waterloo +1

  21. DeALOG: Decentralized Multi-Agents Log-Mediated Reasoning Framework

    Feb 1, 2026Abhijit Chakraborty, Ashish Raj Shekhar, Shiven Agarwal +1Multi-Agent CollaborationMultimodal QA

    Arizona State University

  22. Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis

    Jan 31, 2026Zicheng Kong, Dehua Ma, Zhenbo Xu +9Reward ModelingRL for Language Models

    Beijing University of Posts and Telecommunications, Beijing, P.R. China · ShiFang Technology Inc., Hangzhou, P.R. China · Tsinghua University, Beijing, P.R. China +1

  23. TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs

    Jan 30, 2026Baiqi Li, Kangyi Zhao, Ce Zhang +3VLM EvaluationVideo Understanding

    University of North Carolina at Chapel Hill · University of Pittsburgh · Carnegie Mellon University

  24. SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding

    Jan 29, 2026Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum +2Audio-Language Model EvaluationMultimodal Grounding

    Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON M5G 1L7, Canada · University of Groningen, Nijenborgh 4, 9747 AG Groningen, Netherlands · York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada

  25. AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking

    Jan 25, 2026Xilin Jiang, Qiaolin Wang, Junkai Wu +30Audio-Visual UnderstandingMultimodal Large Language Models

    Columbia University · University of Washington · University of Illinois Urbana-Champaign +5

  26. Social Caption: Evaluating Social Understanding in Multimodal Models

    Jan 21, 2026Leena Mathur, Bhaavanaa Thumu, Youssouf Kebe +1Multimodal Large Language ModelsMultimodal Reasoning

    School of Computer Science, Carnegie Mellon University

  27. CASHEW: Stabilizing Multimodal Reasoning via Iterative Trajectory Aggregation

    Jan 12, 2026Chaoyu Li, Fei Tao, Pooyan FazliTest-Time Scaling for VLMsTest-Time Scaling

    Arizona State University · NewsBreak

  28. NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding

    Jan 3, 2026Hyeonjeong Ha, Jinjin Ge, Bo Feng +2VLM EvaluationVideo Understanding

    Apple · University of Illinois Urbana-Champaign

  29. VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing

    Dec 12, 2025Emanuel Sánchez Aimar, Gulnaz Zhambulova, Fahad Shahbaz Khan +2Cross-Modal LearningMultimodal Embedding

    Linköping University, Sweden

  30. VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs

    Nov 25, 2025Tianxiang Jiang, Sheng Xia, Yicheng Xu +5VLM EvaluationMultimodal Large Language Models

    University of Science and Technology of China · 2Shanghai AI Laboratory · 3Nanjing University +2

  31. SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards

    Nov 10, 2025Hunar Batra, Haoqin Tu, Hardy Chen +3Visual Question AnsweringScene Graph Generation

    University of Oxford · University of California, Santa Cruz

  32. M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset

    Oct 27, 2025Jiahui Geng, Jonathan Tonglet, Iryna GurevychAutomated Fact-CheckingMultimodal Understanding

    Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany · Department of Electrical Engineering, KU Leuven, Belgium +2

  33. From Charts to Code: A Hierarchical Benchmark for Multimodal Models

    Oct 20, 2025Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu +8Data VisualizationCode Generation

    CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University

  34. Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models

    Sep 26, 2025Jiawei Liang, Jianjie Huang, Ruoyu Chen +4Feature AttributionMultimodal Large Language Models

    Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China · Zhongguan-cun Academy, Beijing 100094, China · University of Chinese Academy of Sciences, Beijing 100049, China +2

  35. Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation

    Sep 23, 2025Shufan Li, Jiuxiang Gu, Kangning Liu +4Masked Diffusion ModelsUnified Multimodal Models

    Adobe · UCLA

  36. Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy

    Sep 22, 2025Geewook Kim, Minjoon SeoVLM EvaluationAudio-Visual Understanding

    NAVER Cloud AI, South Korea · KAIST AI, South Korea

  37. Reconstruction Alignment Improves Unified Multimodal Models

    Sep 8, 2025Ji Xie, Trevor Darrell, Luke Zettlemoyer +1Unified Multimodal ModelsCross-Modal Alignment

    1UC Berkeley · University of Washington · 3Duke University

  38. VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark

    Aug 19, 2025Vy Tuong Dang, An Vo, Emilio Villa-Cueva +4VLM EvaluationMultimodal Reasoning

    KAIST · MBZUAI

  39. Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping

    Aug 17, 2025Xuhui Zhan, Tyler DerrCross-Modal LearningCross-Modal Representation Learning

    Data Science Institute, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States. · Computer Science Department, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States.

  40. Position: Reasoning After Perception Means Reasoning Without Vision

    Jul 21, 2025Hongcheng Gao, Zihao Huang, Jingyi Tang +12Visual ReasoningVLM Reasoning

    Tsinghua University · BIT · Peking University +6

  41. Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

    Jun 8, 2025Weiwen Xu, Hou Pong Chan, Long Li +15Medical Report GenerationMultimodal Pretraining

    LASA Team · DAMO Academy, Alibaba Group

  42. LaViDa: A Large Diffusion Language Model for Multimodal Understanding

    May 22, 2025Shufan Li, Konstantinos Kallidromitis, Hritik Bansal +7Vision-Language ModelsEfficient VLM Inference

    1UCLA · 2Panasonic AI Research · 4Salesforce Research +1

  43. HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation

    May 16, 2025Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie +6Algorithmic FairnessMultimodal Large Language Models

    Vector Institute for Artificial Intelligence, Toronto, Canada · University of Central Florida, Orlando, USA

  44. ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

    Feb 13, 2025Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma +31Visual ReasoningVLM Reasoning

    University of Cambridge

  45. MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis

    Oct 19, 2024Junda Wang, Zonghai Yao, Yujan Ting +5HealthcareMedical Image Analysis

    United Imaging Intelligence, MA, USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, MA, USA · Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA +1

  46. VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models

    Jul 16, 2024Haodong Duan, Xinyu Fang, Junming Yang +41VLM EvaluationLLM Evaluation

  47. MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

    Date pendingSangyun Chung, Se Yeon Kim, Youngchae Chee +1Multimodal RobustnessHallucination in Language Models

    Integrated Vision Language Lab, KAIST, South Korea

  48. V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval

    Date pendingDongyang Chen, Chaoyang Wang, Dezhao Su +6Multimodal IRMultimodal Grounding

    Tsinghua University · University of Central Florida · Fudan University +4

  49. Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding

    Date pendingHatice Merve Vural, Doga Kukul, Ege Erdem Ozlu +4VLM ReasoningMultimodal CoT Reasoning

    Koç University, Istanbul, Turkey · KUIS AI Center, Istanbul, Turkey · Air Mail and Cartoon Collections +1

  50. Slot2Text: Object-Centric Visual Tokenization for Efficient and Spatially Traceable Surgical MLLMs

    Date pendingGuiqiu Liao, Matjaz Jogan, Daniel A. HashimotoImage TokenizationHealthcare

    GRASP Laboratory, University of Pennsylvania · PCASO Laboratory, Department of Surgery, University of Pennsylvania · Department of Computer and Information Science, University of Pennsylvania

  51. CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery

    Date pendingMahir Shahriar Tamim, Sharjil Khan, Md. Samiul Alim +3Cross-Modal LearningModality Imbalance

    Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh

  52. Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancy

    Date pendingYi-Cheng Lai, Hen-Hsen HuangLLM SycophancyEvidence-Grounded Reasoning

    Institute of Information Science, Academia Sinica Taipei, Taiwan

  53. Video2Reaction: Training Foundation Video Models to Predict Audience Reaction

    Date pendingSidong Zhang, Trang Nguyen, Shiv Shankar +4VLM AdaptationVideo-Language Models

    UMass Amherst 130 Governors Drive, Amherst · Dolby Laboratories 1275 Market Street, San Francisco