Multimodal Grounding

Follow this topic Ask about this topic

Multimodal grounding studies how AI systems connect words and claims to evidence in images, video, audio, documents, and physical environments. Common approaches include predicting boxes or masks, aligning attention with visual regions, building object or scene graphs, and retrieving evidence for step-by-step reasoning; newer systems also use tools and verifiable rewards to check those links. The work supports applications such as visual question answering, video search, document analysis, GUI interaction, and robotics, where identifying the right evidence matters as much as producing a plausible answer. A central challenge is that fluent, accurate-looking answers may not depend on the relevant evidence, while precise spatial and temporal grounding and reliable use of multiple modalities remain difficult to evaluate and improve.

Latest papers

186 papers21 papers in the last four weeks, down 12% on the four weeks before. 0.1% of all new papers.

Complementing Text Based Moderation with Multimodal Grounding: Detecting Fraudulent Inducement Behaviors on Short Video Platforms
Oct 28, 2026Xiaoyong YuVideo Content Moderation Multimodal Content Moderation
Huazhong University of Science and Technology, Wuhan, China
As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does
Oct 27, 2026Zhongrui Gui, Junyu Xie, Tengda Han +2Video-Language Models Multimodal Grounding
Visual Geometry Group, Department of Engineering Science, University of Oxford, UK · School of Artificial Intelligence (SAI), Shanghai Jiao Tong University, China
RVG-Bench: A Benchmark for Reasoning-Driven Audio-Visual Temporal Grounding
Oct 23, 2026Yue Feng, Boshen Xu, Zihao Yue +5Temporal Video Grounding Multimodal Grounding
AIM3 Lab, Renmin University of China · Kling AI
When More Metadata Misleads: Probing Field-Level Grounding in LLM-Generated Museum Descriptions
Oct 22, 2026Clemencia Siro, Laura HollinkGrounded Text Generation LLM Grounding
Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands
R4DGS: Referring Segmentation in 4D Gaussian Splatting
Oct 21, 2026Bangpu Chen, Yaxuan Li, Shirui Peng +5Multimodal Grounding 4D Scene Understanding
China University of Geosciences Wuhan, China · Xinjiang University Urumqi, China · University of Nottingham Ningbo China Ningbo, China +1
Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding
Oct 21, 2026Xin Zhang, Haochen Wang, Beibei Lin +4Multimodal Grounding Multimodal Understanding
RACE: Learning Multimodal Search through Referential Abstraction and Compositional Evidence
Oct 20, 2026Eric Tao, Minhao Sun, Liyan Liu +9Multimodal Search Agents Multimodal Grounding
Huawei · Beijing Institute of Technology
ACE-Brain-0.6: Scaling Spatial Grounding for Physical Foundation Models
Oct 19, 2026Yinhe Wang, Tao Tao, Zexin He +11Multimodal Grounding Visual Grounding for Robotics
OmniGround: Strengthening Omni-Modal Temporal Grounding via Attention-Consistency Training
Oct 19, 2026Zixuan Chen, Depeng Wang, Hao Lin +5Temporal Video Grounding Multimodal Grounding
Shanghai Jiao Tong University · Ant Group · Independent Researcher
Think, Compress, Answer: Reinforcement Learning for Multimodal Grounding via Reasoning Compression
Oct 19, 2026Jinbae Seo, Hyeongjun Kwon, Jiyoung Lee +4Multimodal Grounding Multimodal Large Language Models
Yonsei University · School of AI and Software, Ewha Womans University · LIG Defense & Aerospace +1
ANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark
Oct 16, 2026Joel Julin, Souraja Kundu, Liza Dahiya +6Multimodal Model Evaluation Multimodal Grounding
Carnegie Mellon University · Amazon
GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning
Oct 13, 2026Qingsong Xie, Jiahao Tian, Jiacheng Tang +5GUI Grounding Multimodal Grounding
Prentis AI, San Francisco, CA · Westlake University, Hangzhou, China · Titan Holdings, San Francisco, CA
OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding
Oct 13, 2026Xuyang Liu, Shihao Wang, Xinyu Wei +5Video-Language Model Evaluation Multimodal Grounding
The Hong Kong Polytechnic University · OPPO Research Institute · Colorado School of Mines +2
From Sound to Sense: An Anatomy of Semantic Supervision for Speech-Driven Visual Grounding in Remote Sensing Imagery
Oct 12, 2026Zhongyu PangConnectionist Temporal Classification Multimodal Grounding
College of Electronic Science, National University of Defense Technology, Changsha 410073, China
GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?
Oct 10, 2026Zonghuan Li, Hayley Hung, Chirag RamanMultimodal Model Evaluation Multimodal Understanding
Department of EEMCS, Delft University of Technology, Delft, Netherlands
EmoGavel: Grounding-Aware Reinforcement Learning with Verifiable Rewards for Multimodal Emotion Preference Learning
Oct 9, 2026Zilong Huang, Kong Aik Lee, Zezhong Jin +2Pairwise Preference Learning Reinforcement Learning with Verifiable Rewards
The Hong Kong Polytechnic University, Hong Kong SAR, China · Kyoto University, Japan
ConClick: Element-Wise Contrastive Learning Improves GUI Grounding
Oct 9, 2026Ruotian Zhang, Yuhui Zhang, Junzhe Huang +3GUI Grounding GUI Agents
GeoReform: Reflective Formalization Evolution for Multimodal Geometry Problem Solving
Oct 8, 2026Jialu Wang, Ruichen Zhang, Xiaoou Liu +2Multimodal Reasoning Multimodal Large Language Models
Tongji University · University of North Carolina at Chapel Hill · Arizona State University
GroundSight at GroundLM 2026 Shared Tasks: GoldenViewVQA
Oct 8, 2026Kun Wang, Yupeng Hu, Ruping Cao +4Self-Correction in VLMs Visual Question Answering
School of Software, Shandong University, Jinan, China · School of Computing, National University of Singapore, Singapore · School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, China
TiTok: Audio-Visual LLM for Multi-Segment Temporal Grounding
Oct 7, 2026Eunji Shin, Dahyun Choi, Seungyeon Jo +2Multimodal Grounding Multimodal Large Language Models
Department of Artificial Intelligence, Ewha Womans University · Division of Artificial Intelligence & Software, Ewha Womans University
A Benchmark for Spatially Grounded Gesture Generation
Oct 2, 2026Anna Deichler, Rishabh Dabral, Fethiye Irmak Dogan +2Co-Speech Gesture Generation Multimodal Grounding
KTH Royal Institute of Technology · MPI for Informatics · University of Cambridge
Mutual Equilibrium: Multimodal Representation Learning through Reciprocal Feedback
Sep 30, 2026Ho-min Park, Byungkon KangMultimodal Grounding Multimodal Embedding
Data Science Center, Texas Children’s Hospital Baylor College of Medicine Houston, Texas 77030, USA · Department of Computer Science SUNY Korea Incheon, Republic of Korea
MG-Thinker: Bi-Axial Self-Reflection for Multi-Image Reasoning Grounding
Sep 29, 2026Heyu Huang, Chi Chen, Zonghao Guo +3Multimodal Grounding Multimodal CoT Reasoning
Huazhong University of Science and Technology Wuhan, China · Tsinghua University Beijing, China
MoGround: Measuring and Mitigating Modality Distraction in Vision-Language Models
Sep 27, 2026Luca Zhou, Bo Zhao, Rose Yu +2VLM Evaluation Multimodal Grounding
Sapienza University of Rome · Harvard University · UC San Diego +2
PhysAlign: A Benchmark for Evidence-Grounded Role Alignment in Multimodal Physics Reasoning
Sep 27, 2026Kecheng Liang, Haoyang Liu, Zexin Chen +5Multimodal Grounding Multimodal Reasoning
Sun Yat-sen University · Dalian University of Technology · Xi’an Jiaotong-Liverpool University
EngIntervene: Benchmarking Multimodal Engineering State Understanding and Design Intervention Reasoning
Sep 27, 2026Jinchang Zhang, Yingda Tao, Jiakai Lin +1Multimodal Grounding Multimodal Reasoning
Intelligent Vision and Sensing (IVS) Lab Indiana University Bloomington Bloomington, IN, USA
Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weighting
Sep 25, 2026Paweł Mąka, Piotr Andruszkiewicz, Yusuf Can Semerci +2Multimodal MT Machine Translation
Department of Advanced Computing Sciences, Maastricht University · Warsaw University of Technology · IDEAS Research Institute
Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration
Sep 24, 2026Jiaqi Deng, Zonghan Wu, Zhan Heng +3Multimodal Grounding Multimodal Large Language Models
University of Technology Sydney · East China Normal University · The University of New South Wales +2
OmniFysics-Nano-V2 Technical Report: Understanding the Physical World Across Modalities
Sep 22, 2026Yizhou Liu, Jinghang Han, Kaixiang Qiu +8Multimodal Grounding Physical Reasoning
Physical Superintelligence Lab, Fysics AI · College of Intelligent Robotics and Advanced Manufacturing, Fudan University
MIGU: Multimodal Instruction Grounding under Uncertainty for Manipulation Planning
Sep 21, 2026Mingke Lu, Anxing Xiao, David HsuMultimodal Grounding Robot Task Planning
School of Computing and Smart Systems Institute, National University of Singapore, Singapore · Department of Electrical and Computer Engineering, University of California, Los Angeles, Los Angeles, CA, USA

Oct 28, 2026J/K move · Enter open · S save

Complementing Text Based Moderation with Multimodal Grounding: Detecting Fraudulent Inducement Behaviors on Short Video Platforms

Xiaoyong Yu

Huazhong University of Science and Technology, Wuhan, China

Video Content Moderation Multimodal Content Moderation Multimodal Grounding

Open paper AskPDF arXiv

Short video platforms have become a primary channel for black and gray industry operations in internet finance, where inducements such as prompts to like, follow, join private chat groups, or click external links frequently steer users toward fraud. Because such videos are short, dynamic, and often convey intent through fleeting visual cues rather than stable text, a moderation system must guarantee detection recall and localize the offending segment so reviewers inspect only the relevant clip. Existing OCR+LLM pipelines transcribe on-screen text frame by frame and classify the transcript, but exploit only textual information. Many inducements are conveyed through visual cues OCR cannot observe, and keyframe-based localization cannot recover precise temporal boundaries. We explore OCR+MLLM, which augments the text pipeline with a Multimodal Large Language Model, and compare three routes: OCR+LLM, OCR+MLLM, and a large zero-shot MLLM, along detection, latency, token consumption, and deployment complexity. OCR preserves the text signal that guarantees recall, while the MLLM jointly performs OCR-grounded discrimination and temporal grounding, localizing both textual and non-textual inducements. We construct a dataset of 2,419 internet finance short videos (66.49 hours) with 3,194 annotated events across four categories, and study reward design for RL-based grounding under class imbalance. We report two findings: auxiliary reward terms are exploited through reward hacking, and stronger localization rewards degrade class-balanced detection. We therefore design a recall-weighted F2 localization reward with class-balanced resampling. Experiments show OCR+MLLM localizes better than the keyframe OCR+LLM baseline (mIoU 26.52 vs. 14.49) and detects non-textual inducements that text-only pipelines miss, while maintaining 67.0 percent macro recall.