Multimodal Grounding

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 122

All topics

MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild
Jun 15, 2026Haotian Qi, Gabriel SkantzeMultimodal Grounding Audio-Visual Understanding
Department of Speech Music and Hearing, KTH Stockholm, Sweden
Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning
Jun 13, 2026Zhengbo Zhang, Changtao Miao, Jinbo Su +10Multimodal IR Multimodal Grounding
School of Artificial Intelligence, UCAS · Institute of Automation, CAS · Ant Digital Technologies, Ant Group +2
Findings of the MAGMaR 2026 Shared Task
Jun 10, 2026Alexander Martin, Dengjia Zhang, Joel Brogan +7Multimodal RAG Multimodal Grounding
Johns Hopkins University · OpenAI · University of Maryland, Baltimore County +4
Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks
Jun 9, 2026Miaoxin Cai, Guanqun Wang, Wei Zhang +6Remote Sensing Image Understanding Multimodal Grounding
National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing (SBIIP), Beijing Institute of Technology, Beijing 100081, China · Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology, Beijing 100081, China · School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China +5
VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?
Jun 5, 2026Didi Zhu, Changrui Chen, Stefanos Zafeiriou +1VLM Evaluation Multimodal Grounding
Imperial College London
M $^3$ Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions
Jun 5, 2026Zhengjun Huang, Wenxuan Liu, Zhoujin Tian +6Multimodal Grounding Efficient Multimodal Inference
The Hong Kong University of Science and Technology · Peng Cheng Laboratory · Beijing University of Chemical Technology +4
FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
Jun 2, 2026Eshika Khandelwal, Jingjing Pan, Mingfang Zhang +3Multimodal Grounding Video Object Detection
Tuebingen AI Center, University of Tuebingen · Woven by Toyota, Inc., Tokyo, Japan · Toyota Motor Europe, Brussels, Belgium
The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialogue
Jun 1, 2026Sherzod Hakimov, Mattia D'Agostini, Ivan Samodelkin +1VLM Evaluation Multimodal Grounding
Computational Linguistics, Department of Linguistics University of Potsdam, Germany · German Research Center for Artificial Intelligence (DFKI), Berlin, Germany
Agent Skills Should Go Beyond Text: The Case for Visual Skills
May 31, 2026Binxiao Xu, Ruichuan An, Bocheng Zou +1Multimodal Grounding Agent Skill Learning
Peking University · University of Wisconsin · MIT-IBM Watson AI Lab
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
May 31, 2026Wanlong Fang, Tianle Zhang, Wen Tao +1Multimodal Grounding Multimodal Large Language Models
College of Computing and Data Science, Nanyang Technological University, Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore · Centre of AI in Medicine (C-AIM), Nanyang Technological University, Singapore
TIGER: Traceable Inference with Graph-Based Evidence Routing for Mitigating Hallucinations in Multimodal Generation
May 29, 2026Kaixiang Zhao, Tianrun Yu, Shawn Huang +3Multimodal Grounding Multimodal Generation
Brigham Young University · Florida State University
Recognizing Co-Speech Gestures in-the-Wild
May 29, 2026Sindhu B Hegde, K R Prajwal, Andrew ZissermanMultimodal Grounding Gesture Recognition
Visual Geometry Group, Dept. of Engineering Science, University of Oxford
GUI-C $^2$ : Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning
May 29, 2026Junlong Li, Chao Hao, Lap-Pui Chau +1Multimodal Grounding Agentic RL
The Hong Kong Polytechnic University
REVEAL: Reference-Grounded Reasoning for Multimodal Manipulation Detection
May 27, 2026Jun Zhou, Bingwen Hu, Yaxiong Wang +4Multimodal Grounding Image Forgery Detection
ROVER: Routing Object-Centric Visual Evidence for Grounded Multi-Image Reasoning
May 27, 2026Guannan Lv, Ren Nie, Hongjian Dou +1Multimodal Grounding Efficient Multimodal Inference
Kuaishou Technology
Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization
May 27, 2026Jiawei Kong, Hao Fang, Shunxiang Liao +5Multimodal Grounding Direct Preference Optimization
Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology, Shenzhen
MinhwaNet: Faithful but Insufficient Object Grounding in Korean Folk Painting
May 26, 2026Joonhyung BaeMultimodal Grounding Multimodal Classification
Korea Advanced Institute of Science and Technology (KAIST), South Korea · Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea
AnchorDiff: Training-Free Concept Grounding for MM-DiTs via Anchor-Based Graph Propagation
May 26, 2026Jian Zhang, Zhijun ZhangMultimodal Grounding Multimodal Diffusion Models
School of Automation Science and Engineering South China University of Technology Guang Zhou, China
Advancing Creative Physical Intelligence in Large Multimodal Models
May 25, 2026Cheng Qian, Hyeonjeong Ha, Jiayu Liu +10Multimodal Grounding Object Hallucination in VLMs
1UIUC · 2Amazon
DuoGesture: Motion-Grounded Semantic Conditioning and Biomechanical Beat Priors for Co-Speech Gesture Generation
May 25, 2026Ferdinand Paar, Lanmiao Liu, Aslı Özyürek +2Multimodal Grounding Human Motion Generation
Radboud University, Nijmegen · Max Planck Institute for Psycholinguistics · Utrecht University
PoseRefer: Pathway-Local Parameters for Semantically Grounded Reference Resolution
May 23, 2026Anna DeichlerMultimodal Grounding Multimodal Fusion
KTH Royal Institute of Technology, Stockholm, Sweden
GesVLA: Gesture-Aware Vision-Language-Action Model Embedded Representations
May 21, 2026Wenxuan Guo, Ziyuan Li, Meng Zhang +7Multimodal Grounding Robotic Manipulation
1Tsinghua University · 2Dexmal
HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding
May 19, 2026Mengqi Shi, Haopeng ZhangMultimodal Grounding Video Summarization
Department of Information and Computer Sciences · University of Hawaii at Manoa
TIGER-FG: Text-Guided Implicit Fine-Grained Grounding for E-commerce Retrieval
May 18, 2026Xinyu Sun, Huangyu Dai, Lingtao Mao +5Multimodal IR Multimodal Grounding
Kuaishou Technology · Beijing, China
IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning
May 18, 2026Chenghao Li, Fusheng Hao, Xikai Zhang +5Multimodal Grounding Object Hallucination in VLMs
Hangzhou International Innovation Institute, Beihang University · Shenzhen Institute of Advanced Integration Technology, Shenzhen · Kuaishou Technology +1
From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAG
May 14, 2026Guanhua Chen, Chuyue Huang, Yutong Yao +4Multimodal RAG Multimodal Grounding
NLP2CT Lab, Department of Computer and Information Science, University of Macau
Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs
May 13, 2026Trung Nguyen Quang, Yiming Gao, Fanyi Pu +3Multimodal Grounding Multimodal Large Language Models
Nanyang Technological University · LMMs-Lab Team · Johns Hopkins University
Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
May 13, 2026Qinwu Xu, Yifan Jiang, Haoyu RenMultimodal Grounding CoT Reasoning
Meta AI · UT Austin · Current independent researcher
ViDR: Grounding Multimodal Deep Research Reports in Source Visual Evidence
May 13, 2026Zhuofan Shi, Peilun Jia, Baoqin Sun +4Multimodal RAG Multimodal Grounding
School of Software and Microelectronics, Peking University, Beijing 100871, China · 2National Key Laboratory of Data Space Technology and System, Beijing 100871, China · School of Software Engineering, Beijing Jiaotong University, Beijing, 100044, China
RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild
May 11, 2026Danni Xu, Shaojing Fan, Harry Cheng +1VLM Evaluation Multimodal Grounding
School of Computing (SoC), National University of Singapore (NUS), Singapore · Department of Electrical and Computer Engineering (ECE), National University of Singapore (NUS), Singapore

Jun 15, 2026, cs.SDJ/K move · Enter open · S save

MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild

Haotian Qi, Gabriel Skantze

Department of Speech Music and Hearing, KTH Stockholm, Sweden

Multimodal Grounding Audio-Visual Understanding Turn-Taking Prediction Turn-Taking in Dialogue

Open paper AskPDF arXiv

Current multiparty turn-taking models often rely on complex microphone arrays or multi-camera setups, limiting their applicability in human-robot interaction scenarios. We introduce MuVAP, a causal multimodal framework that extends Voice Activity Projection by grounding acoustic predictions in face tracks, enabling speaker-aware turn-taking predictions from a monaural audio stream and a single camera view. To address the combinatorial complexity of modeling multiple speakers, we propose Role-Relative Projection, which maps any N-speaker interaction onto a fixed current versus next floor-holder state. Because existing audiovisual datasets contain disruptive editing cuts that break causal tracking, we introduce the Audio-Visual Conversation Corpus, a 31-hour dataset of unedited, single-camera multiparty conversations. Evaluations demonstrate that MuVAP outperforms strong baselines on Shift-Hold and next-speaker prediction tasks across two- and three-speaker settings.

Momentum

Related topics

Latest papers 122