Multimodal Grounding

Ask about this topic

Overview

Multimodal grounding studies how AI systems connect words and claims to evidence in images, video, audio, documents, and physical environments. Common approaches include predicting boxes or masks, aligning attention with visual regions, building object or scene graphs, and retrieving evidence for step-by-step reasoning; newer systems also use tools and verifiable rewards to check those links. The work supports applications such as visual question answering, video search, document analysis, GUI interaction, and robotics, where identifying the right evidence matters as much as producing a plausible answer. A central challenge is that fluent, accurate-looking answers may not depend on the relevant evidence, while precise spatial and temporal grounding and reliable use of multiple modalities remain difficult to evaluate and improve.

Latest papers

CardsList

186 papers21 papers in the last four weeks, down 12% on the four weeks before. 0.1% of all new papers.

  1. LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation

    Apr 13, 2026Yuqian Yuan, Wenqiao Zhang, Juekai Lin +7Multimodal Large Language ModelsObject-Centric Representation Learning

    Zhejiang University · Nanjing University of Aeronautics and Astronautics

  2. Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generation

    Apr 12, 2026Fangda Ye, Zhifei Xie, Yuxin Hu +5Multimodal GenerationMultimodal Grounding

    National University of Singapore · Nanyang Technological University · University of Edinburgh +1

  3. GeoMeld: Toward Semantically Grounded Foundation Models for Remote Sensing

    Apr 12, 2026Maram Hasan, Md Aminur Hossain, Savitra Roy +6Multimodal Foundation ModelsMultimodal Grounding

    Indian Institute of Technology Bombay · Space Applications Centre, ISRO · University of Trento +1

  4. STORM: End-to-End Referring Multi-Object Tracking in Videos

    Apr 12, 2026Zijia Lu, Jingru Yi, Jue Wang +4Referring MOTMulti-Object Tracking

    Amazon

  5. Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimization

    Apr 9, 2026Sai Srinivas Kancheti, Aditya Kanade, Rohit Sinha +2Visual Spatial ReasoningMultimodal CoT Reasoning

    CSE Dept., IIT Hyderabad, Hyderabad, India · Microsoft Research, Bengaluru, India

  6. Tree-of-Evidence: Efficient "System 2" Search for Faithful Multimodal Grounding

    Apr 9, 2026Micky C. Nnamdi, Benoit L. Marteau, Yishan Zhong +2Multimodal Model InterpretabilityMultimodal Grounding

    Georgia Institute of Technology

  7. FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios

    Apr 8, 2026Xiangru Jian, Hao Xu, Wei Pang +13Multimodal Model EvaluationMultimodal Learning

    University of Waterloo, Canada · University of Sydney, Australia · Singapore Management University, Singapore +7

  8. PanopticQuery: Unified Query-Time Reasoning for 4D Scenes

    Apr 7, 2026Ruilin Tang, Yang Zhou, Zhong Ye +34D Scene UnderstandingMultimodal Grounding

    School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · School of Computing and Information Systems, Singapore Management University, Singapore 188065 · School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510006, China +1

  9. Simulating Validity: Modal Decoupling in MLLM Generated Feedback on Science Drawings

    Apr 5, 2026Arne Bewersdorff, Nejla Yuruk, Xiaoming ZhaiScience EducationMultimodal Grounding

    University of Georgia, AI4STEM Education Center, Athens GA 30602, USA · Gazi University, Department of Mathematics and Science Education, Ankara, 06560, Turkiye

  10. Communicating about Space: Language-Mediated Spatial Integration Across Partial Views

    Mar 28, 2026Ankur Sikarwar, Debangan Mishra, Sudarshan Nikhil +2Spatial Reasoning in Language ModelsSpatial Reasoning

    Mila – Quebec AI Institute · Université de Montréal · IIIT Hyderabad +1

  11. Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision

    Mar 27, 2026Ling Li, Bowen Liu, Zinuo Zhan +4Multimodal GroundingVisual Grounding

    Tsinghua University, Beijing, China · Dalian University of Technology, Dalian, China · Northwestern Polytechnical University, Xi’an, China +1

  12. SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matching

    Mar 27, 2026Lanmiao Liu, Esam Ghaleb, Aslı Özyürek +1Multimodal GroundingFlow Matching

    Max Planck Institute for Psycholinguistics · Donders Institute for Brain Cognition and Behaviour · Utrecht University

  13. Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs

    Mar 26, 2026Vishal Narnaware, Animesh Gupta, Kevin Zhai +2Language Model DecodingMultimodal Grounding

    Institute of Artificial Intelligence, University of Central Florida

  14. OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Models

    Mar 25, 2026Seunghee Kim, Bumkyu Park, Kyudan Jung +5Multimodal GroundingUnified Multimodal Models

    Hanyang University · Seoul National University · KAIST AI +1

  15. MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI

    Mar 20, 2026Rozain Shakeel, Abdul Rahman Mohammad Ali, Muneeb Mushtaq +2GUI GroundingMultimodal Grounding

    Gaash Research Lab, National Institute of Technology Srinagar, India · e& Group, UAE · Mohammad Bin Zayed University of Artificial Intelligence (MBZUAI), UAE +1

  16. From Instructions to Assistance: a Dataset Aligning Instruction Manuals with Assembly Videos for Evaluating Multimodal LLMs

    Mar 20, 2026Federico Toschi, Nicolò Brunello, Andrea Sassella +2Multimodal Large Language ModelsMultimodal Model Evaluation

    DEIB, Politecnico di Milano, Via Ponzio 34/5, 20133, Milano (MI), Italy · KASTEL, Karlsruhe Institute of Technology, Am Fasanengarten, 576131, Karlsruhe, Germany

  17. Enhancing Alignment for Unified Multimodal Models via Semantically-Grounded Supervision

    Mar 20, 2026Jiyeong Kim, Yerim So, Hyesong Choi +2Unified Multimodal ModelsMultimodal Generation

    Ewha Womans University, South Korea · Soongsil University, South Korea

  18. A Proposal-Free Query-Guided Network for Grounded Multimodal Named Entity Recognition

    Mar 18, 2026Hongbing Li, Jiamin Liu, Shuo Zhang +1Named Entity RecognitionMultimodal Grounding

    School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China

  19. Logics-Parsing-Omni Technical Report

    Mar 10, 2026Xin An, Jingyi Cai, Xiangyang Chen +22Multimodal UnderstandingSemantic Parsing

    Alibaba Group

  20. MJ1: Multimodal Judgment via Grounded Verification

    Mar 9, 2026Bhavesh Kumar, Dylan Feng, Leonard TangPosition Bias in VLMsMLLM Evaluation

    Haize Labs

  21. Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time

    Mar 9, 2026Weijie Zhou, Xuantang Xiong, Zhenlin Hu +6Multimodal GroundingAudio-Visual Alignment

    Beijing Jiaotong University · Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences (CASIA) · Tencent Robotics X +2

  22. BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence

    Mar 9, 2026Biao Xiang, Soyeon Caren Han, Yihao DingMulti-Hop QAMultimodal QA

    The University of Melbourne Melbourne, Australia · The University of Western Australia Perth, Australia

  23. DocCogito: Aligning Layout Cognition and Step-Level Grounded Reasoning for Document Understanding

    Mar 8, 2026Yuchuan Wu, Minghan Zhuo, Teng Fu +3Document UnderstandingMultimodal Large Language Models

    Fudan University, Shanghai, China

  24. Seeing the Context: Rich Visual Context-Aware Speech Recognition via Multimodal Reasoning

    Mar 7, 2026Wenjie Tian, Mingchen Shao, Bingshen Mu +8Audio-Visual Speech RecognitionMultimodal CoT Reasoning

    Northwestern Polytechnical University, Xi’an, China

  25. ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning

    Mar 4, 2026Yiting Chen, Kenneth Kimble, Edward H. Adelson +20Robotic ManipulationRobot Manipulation Benchmarks

    Rice University · U.S. National Institute of Standards and Technology · Massachusetts Institute of Technology +12

  26. ECHO: Event-Centric Hypergraph Operations via Multi-Agent Collaboration for Multimedia Event Extraction

    Mar 4, 2026Hailong Chu, Hongbing Li, Yunlong Chu +6Event ExtractionMultimodal Grounding

    Beijing University of Posts and Telecommunications Beijing, China · Tianjin University Tianjin, China · Chongqing University of Posts and Telecommunications Chongqing, China +1

  27. Graph-of-Mark: Promote Spatial Reasoning in Multimodal Language Models with Graph-Based Visual Prompting

    Mar 2, 2026Giacomo Frisoni, Lorenzo Molfetta, Mattia Buzzoni +1Spatial Reasoning in Language ModelsVLM Reasoning

    Department of Computer Science and Engineering, University of Bologna

  28. CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Framework

    Mar 2, 2026Yuexi Du, Jinglu Wang, Shujie Liu +2Medical VLMsMedical Reasoning

    Microsoft Research Asia · Department of Biomedical Engineering, Yale University · Department of Radiology & Biomedical Imaging, Yale University

  29. InterCoG: Towards Spatially Precise Image Editing with Interleaved Chain-of-Grounding Reasoning

    Mar 2, 2026Yecong Wan, Fan Li, Chunwei Wang +3Image EditingText-Guided Image Editing

    Faculty of Computing, Harbin Institute of Technology, Harbin, 150001, China. · Zhengzhou Advanced Research Institute of Harbin Institute of Technology, Zhengzhou, 450000, China. · Huawei Noah’s Ark Lab, Shenzhen, 518100, China. +1

  30. A Multimodal Framework for Aligning Human Linguistic Descriptions with Visual Perceptual Data

    Feb 23, 2026Joseph BinghamReferring Expression ComprehensionMultimodal Grounding

    Faculty of Biology, Technion University, Haifa, Israel.