Multimodal Understanding

Ask about this topic

Momentum

10 papers in the last four weeks, up 233% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 413

All topics
CardsList

  1. Gestalt: Large Multimodal Interplay Model

    Sep 30, 2026Zequn Yang, Yu Miao, Haotian Ni +8Unified Multimodal ModelsCross-Modal Alignment

    Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Beihang University +3

  2. PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation

    Sep 29, 2026Cong Wei, Xuanchi Ren, Bryan Chu +8Unified Multimodal ModelsMultimodal Generation

    NVIDIA

  3. Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models

    Sep 29, 2026Shuyang Jiang, Fucheng Deng, Yuchuan Luo +1Unified Multimodal ModelsGradient Interference

    University of California, Los Angeles · Aimakj · College of Computer Science and Technology National University of Defense Technology +1

  4. Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Models

    Sep 28, 2026Wentao Zhou, Weijie Gan, Jiayun WangAdversarial TrainingUnified Multimodal Models

    Georgia Institute of Technology · Washington University in St. Louis

  5. Virtual Encoders in Multimodal Transformers

    Sep 22, 2026Katsuya Ogata, Yuta NakashimaVisual Representation LearningMultimodal Large Language Models

    The University of Osaka

  6. Efficient Unified Multimodal Understanding (EUMU): Winning Solution for the MUMU Track at the 8th LSVOS Challenge

    Sep 16, 2026Dayoung Kil, Seong-heum KimOpen-Vocabulary Object DetectionEfficient Multimodal Inference

    Department of Intelligent Semiconductors, Soongsil University, Seoul, Republic of Korea

  7. Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialogue

    Sep 14, 2026Leon Hammerla, Patrick Schrottenbacher, Alexander MehlerMultimodal Understanding

    Goethe University, Frankfurt am Main, Germany

  8. MultiHuSE: A Multimodal Dataset for Humour Styles and Emotions

    Sep 11, 2026Mary Ogbuka Kenneth, Foaad Khosmood, Abbas EdalatText ClassificationMultimodal Understanding

    Algorithmic Human Development group, Department of Computing Imperial College London London, United Kingdom · Computer Engineering Department California Polytechnic State University San Luis Obispo, United States

  9. DYAD: A Multimodal Dataset of Co-Located Human Assistance

    Sep 8, 2026Akhil Ajikumar, Mahya Qorbani, Sakib Reza +2Human-AI InteractionAssistive Robotics

    Georgia Institute of Technology, Atlanta, GA, USA · Northeastern University, Boston, MA, USA · Microsoft Research, Redmond, WA, USA

  10. BanglaMemeX: Advancing Cultural Metaphoric Image Interpretation in Bangla with a Multimodal Explainable Dataset

    Sep 7, 2026Md. Sadman Sakib, Zisan Mahmud, Md. Fahim Arefin +1Figurative Language UnderstandingMultilingual VLM Evaluation

    University of Dhaka · University of Texas at Dallas

  11. Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

    Sep 1, 2026Penghao Wu, Haiwen Diao, Weichen Fan +3Unified Multimodal ModelsMultimodal Generation

    1S-Lab, Nanyang Technological University · 2SenseTime Research

  12. Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction

    Aug 12, 2026Zhongbin Guo, Jiahao Xie, Dongling Xiao +5Multimodal PretrainingMultimodal Understanding

    ByteDance

  13. Do You See What You Draw? A Semantic Closed-Loop Framework for Holistic Evaluation of Unified Multimodal Models

    Aug 12, 2026Hao Zhang, Jiaxin Qi, Zhijiang Tang +1Unified Multimodal ModelsMultimodal Reasoning

    Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Computer Network Information Center, Chinese Academy of Sciences

  14. LMM Modality Transfer: A Pre-requisite for Autonomous GIS Agents

    Aug 7, 2026Ivan Majic, Zexian Huang, Franziska Hübl +5Cross-Modal AlignmentMultimodal Large Language Models

    Graz University of Technology, Austria · University of Vienna, Austria · University of Liverpool, UK

  15. M3^3R-Bench: A Unified Benchmark for Evidence-Grounded Multimodal Metaphor Understanding

    Aug 6, 2026Hong Jiang, Junnan Zhu, Jingwang Huang +9Figurative Language UnderstandingMultimodal Reasoning

    School of Computer Science, Chongqing University, Chongqing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore +3

  16. Modality Agreement- and Conflict-Aware Prototype Hypergraph Learning for Multimodal Intent Understanding

    Aug 4, 2026Mohnish Raj, Suraj Kumar, Soumi Chattopadhayay +2Prototype LearningMultimodal Fusion

    Mohnish Raj, Suraj Kumar, Graduate Student Member, IEEE, Soumi Chattopadhayay, Senior Member, IEEE, Chandranath Adak, Senior Member, IEEE, Ayan Dutta

  17. Evidence-Grounded Multimodal Knowledge Graph Construction for Multi-Lecture Educational Reasoning

    Aug 4, 2026Sahil Al Farib, Momota Ahsana Meem, Sheikh Redwanul Islam +1KG ConstructionMultimodal KGs

    Department of Computer Science & Engineering, United International University, Dhaka, Bangladesh

  18. CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

    Aug 3, 2026Zhipeng Liu, Haochen Wang, Zhaoxiang ZhangImage CaptioningT2I Generation

    University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences

  19. Where did the ambiguity go? Examining how multimodal models interpret polysemous words

    Aug 1, 2026Jasin Cekinmez, Addison J. Wu, Raja Marjieh +1Multimodal GenerationModality Gap in VLMs

    Princeton University

  20. CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understanding

    Jul 31, 2026Wenxin Tang, Jingyu Xiao, Zhenyu Liu +6Efficient Multimodal InferenceMultimodal Large Language Models

    Tsinghua University · The Chinese University of Hong Kong · Northwestern Polytechnical University +4

  21. FriendBench: Benchmarking Dyadic Familiarity Inference in Humans and Multimodal Large Language Models

    Jul 31, 2026Jeffrey M. Girard, Jason Z. Zheng, Jacqueline R. Vertino +2ClassificationMultimodal Model Evaluation

    Fluid Concepts Research

  22. Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

    Jul 28, 2026Jiaang Li, Chengzu Li, Zhaochong An +4VLM EvaluationMultimodal Large Language Models

    University of Copenhagen · University of Cambridge · 3ETH Zürich +1

  23. Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation

    Jul 28, 2026Weiming Zhuang, Jiabo Huang, Jingtao Li +4Unified Multimodal ModelsVisual Tokenization

    Sony AI

  24. From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligence

    Jul 18, 2026Nadine Chang, Maying Shen, Shizhe Diao +6Multimodal IRCross-Modal Representation Learning

    NVIDIA · University of Ottawa

  25. Monkey King Bang: A Unified Scientific Multimodal Foundation Model

    Jul 17, 2026Hesen Chen, Xinyu Su, Xiaomeng Yang +11AI for ScienceUnified Multimodal Models

    SAIS Team, Shanghai Academy of AI for Science

  26. Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

    Jul 9, 2026Dominick Reilly, Qiyu Wu, Hiromi Wakaki +2Missing-Modality LearningCross-Modal Representation Learning

    Sony Group Corporation · University of North Carolina at Charlotte · Sony AI

  27. SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits

    Jul 9, 2026Adis Alihodzic, Selma Skopljakovic HubljarSHAP Feature AttributionMultimodal Emotion Recognition

    Department of Mathematical and Computer Sciences, Faculty of Science, University of Sarajevo, Sarajevo, Bosnia and Herzegovina.

  28. Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing

    Jul 9, 2026Feng Wang, Canmiao Fu, Zhipeng Huang +3Multimodal MemoryEpisodic Memory

    Peking University · WeChat Vision, Tencent Inc.

  29. DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Models

    Jul 9, 2026Pengjie Wang, Linger Deng, Zujia Zhang +6Efficient Multimodal InferenceUnified Multimodal Models

    Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc.

  30. Reinforcing the Generation Order of Multimodal Masked Diffusion Models

    Jul 9, 2026Yidong Ouyang, Zhe Wang, Sourav Bhabesh +1Text-Image AlignmentRL for Diffusion Models

    University of California, Los Angeles · AGI Foundations for AWS

  31. Infinity-Parser2 Technical Report

    Jul 8, 2026Zuming Huang, Jun Huang, Kexuan Ren +12Synthetic Data GenerationMultimodal Large Language Models

  32. Do You Remember? Toward Memory-Centric Multimodal AI

    Jul 7, 2026Xuguang Yu, Weigang Zheng, Minyue YuMemory-Augmented VLMsMultimodal Memory

  33. Vision as Unified Multimodal Generation

    Jul 7, 2026Xiaoyang Han, Jianhua Li, Kewang Deng +14Unified Multimodal ModelsMultimodal Generation

    SenseTime Research · Peking University · Shanghai Jiao Tong University +3

  34. CMDR: Contextual Multimodal Document Retrieval

    Jul 7, 2026Ryota Tanaka, Taku Hasegawa, Kyosuke NishidaMultimodal IRMultimodal Embedding

    Human Informatics Labs., NTT, Inc., Tokyo, Japan

  35. Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts

    Jul 7, 2026Andrei-George Durdun, Victor Constantinescu, Radu Tudor IonescuCross-Modal LearningSentiment Analysis

    Department of Computer Science, University of Bucharest, 15G Iuliu Maniu, Bucharest 061075, Romania · Department of Data Science, PPC Romania, 30 Mircea Voda, Bucharest 030667, Romania

  36. Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory

    Jul 6, 2026Chang Nie, Jiaju Wei, Junlan Feng +2Multimodal AgentsMemory-Augmented Agents

    Nanjing University

  37. Unified Audio Intelligence Without Regressing on Text Intelligence

    Jul 6, 2026Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim +17TTS SynthesisUnified Multimodal Models

    NVIDIA

  38. Enhancing Large Multimodal Models in Key Information Extraction via Scene-Aware Document Synthesis

    Jul 6, 2026Zhipeng Xu, Zulong Chen, Qing Liu +6Synthetic Data GenerationDocument Information Extraction

    Alibaba Group · Taobao and Tmall Group, Alibaba · Qwen Team, Alibaba Group +1

  39. Hierarchical Evidence-Driven Reasoning for Long Document Understanding

    Jul 6, 2026Junyu Xiong, Yonghui Wang, Rongjian Gu +4Multi-Hop QADocument Understanding

    University of Science and Technology of China · iFLYTEK Research

  40. Transferability Between Understanding and Generation in Unified Multimodal Models

    Jul 5, 2026Jiwon Kang, Heeji Yoon, Jaewoo Jung +5Unified Multimodal ModelsMultimodal Generation

    KAIST AI, South Korea · Blynx, South Korea · Trillionlabs, South Korea

  41. Q-TriM: Question-Guided Tri-Modal Attention for Audio-Visual Question Answering

    Jul 4, 2026SungHun Kim, SeungJun BaekMultimodal QACross-Modal Attention

    Dept. of Computer Science and Engineering Korea University Seoul, South Korea

  42. Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

    Jul 4, 2026Zican Hu, Xuyang Hu, Yiming Liu +10Unified Multimodal ModelsMultimodal Reasoning

    Nanjing University · Shanghai AI Laboratory · Tsinghua University +5

  43. Attending to Multimodal Generation One Token at a Time

    Jul 4, 2026Varun Gupta, Vineet Gandhi, Makarand TapaswiMultimodal Large Language ModelsMultimodal Generation

    CVIT, IIIT Hyderabad

  44. Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG

    Jul 3, 2026Xue Li, Yiming GaiMultimodal RAGEfficient Multimodal Inference

    Hangzhou International Innovation Institute, Beihang University Hangzhou, China

  45. OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models

    Jul 3, 2026Shijie Cao, Qingyu Zhang, Boxi Yu +6Efficient Multimodal InferenceMultimodal Large Language Models

    School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Limerick +1

  46. Gemma 4 Technical Report

    Jul 2, 2026Gemma Team, Sherif El Abd, Vaibhav Aggarwal +298Efficient Multimodal InferenceUnified Multimodal Models

  47. Show Me Examples: Inferring Visual Concepts from Image Sets

    Jul 2, 2026Nick Stracke, Kolja Bauer, Stefan Andreas Baumann +3Vision-Language ModelsConditional Image Generation

    Munich Center for Machine Learning · Apple

  48. Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Fails

    Jul 2, 2026Qianyu Chen, Canran Xiao, Runxuan TangMultimodal GroundingMultimodal Foundation Models

    1Nanyang Technological University · 2Shenzhen Campus of Sun Yat-sen University

  49. Towards High-Resolution Visual Perception via Hierarchical Entity Exploration

    Jul 1, 2026Ziyu Ma, Shidong Yang, Yuxiang Ji +5Large Vision-Language ModelsFine-Grained Visual Perception

    AMAP, Alibaba Group, China · Monash University, Australia

  50. M2Note: Continual Evolution of Vision Language Models via Mistake Notebook Learning

    Jul 1, 2026Haiwen Li, Jing Tang, Rui Chen +2Memory-Augmented VLMsVLM Reasoning

    AMAP, Alibaba Group

  51. H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation

    Jul 1, 2026Qixiang Yin, Huanjin Yao, Yuchen Cai +5Cross-Modal Knowledge DistillationVLM Distillation

    Beijing University of Posts and Telecommunications · Zhongguancun Academy, Beijing, China · ByteDance +3

  52. Homer: Understanding Long-form Videos with Hierarchical Memory and Agentic Reasoning

    Jul 1, 2026Yixin Ji, Fanghua Ye, Juntao Li +5Long-Video UnderstandingMemory-Augmented Video Understanding

    1Soochow University · 2Tencent Hunyuan Multimodal Department

  53. Multimodal Continuous Reasoning via Asymmetric Mutual Variational Learning

    Jul 1, 2026Shijie Li, Yilin Gao, Siyuan Yang +7Multimodal Large Language ModelsLatent Visual Reasoning

    Shanghai Jiao Tong University · Ant Group

  54. Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs

    Jul 1, 2026Yoonhyung Park, Minji Kim, Sungwon Moon +1Visuo-Tactile Representation LearningMultimodal Large Language Models

    Division of Artificial Intelligence & Software, Ewha Womans University

  55. Rosetta: Composable Native Multimodal Pretraining

    Jul 1, 2026Xiangyue Liu, Zijian Zhang, Miles Yang +3Unified Multimodal ModelsMultimodal Pretraining

    HKUST · Tencent Hunyuan

  56. CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts

    Jun 30, 2026Lianyu Hu, Shengqian Qin, Zeqin Liao +4Efficient Multimodal InferenceEfficient Language Model Reasoning

    Nanyang Technological University, Singapore · Shanghai Jiao Tong University, China · NKIARI, Shenzhen Futian, China +2

  57. Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference

    Jun 30, 2026Zhaoyang Luo, Runmin Dong, Miao Yang +4Efficient VLM InferenceMultimodal Large Language Models

    Tsinghua Shenzhen International Graduate School, Shenzhen, China · Sun Yat-sen University, Zhuhai, China · Tsinghua University, Beijing, China +1

  58. Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning

    Jun 30, 2026Kaitao Chen, Weiqian Zhao, Jiamin Wu +6Efficient Multimodal InferenceMedical VQA

    1Fudan University · 2Shanghai Artificial Intelligence Laboratory · 3Shanghai Jiao Tong University +1

  59. MS-Resampler: Multi-Scope Visual Resampling for Efficient Multimodal LLMs

    Jun 30, 2026Zhongyang Li, Yaqian Li, Faming Fang +6Multimodal Large Language ModelsMultimodal Fusion

    Li Auto Inc. · East China Normal University.

  60. Learning to Deny: Action Denial in Multimodal Large Language Models

    Jun 30, 2026Raiyaan Abdullah, Shehreen Azad, Yogesh Singh RawatMultimodal Large Language ModelsCausal Reasoning in Videos

    Institute of Artificial Intelligence, University of Central Florida, USA