Multimodal Understanding

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, up 233% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 413

All topics

HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
May 6, 2026Esra Dönmez, Pascal Tilli, Hsiu-Yu Yang +2Cross-Modal Learning Image Matching
Institute for Natural Language Processing, University of Stuttgart
To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition
May 6, 2026Yangchen Yu, Qian Chen, Jia Li +5Cross-Modal Learning Multimodal Robustness
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China · School of Computing and Information Systems, Singapore Management University, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore +1
UnAC: Adaptive Visual Prompting with Abstraction and Stepwise Checking for Complex Multimodal Reasoning
May 5, 2026Yifan Wang, Yun FuMultimodal Large Language Models Multimodal Reasoning
Northeastern University
JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generation
May 5, 2026Lin Song, Wenbo Li, Guoqing Ma +16Unified Multimodal Models Multimodal Foundation Models
Joy Future Academy, JD
ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval
May 5, 2026Honglei Zhang, Yuting Chen, Chenpeng Hu +2Audio Reasoning Audio Understanding
School of Software, Nanjing University Nanjing, China · School of Software, Northwestern Polytechnical University Xi’an, China · College of Computing and Data Science, Nanyang Technological University Singapore, Singapore
Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts
May 5, 2026Hahyeon Choi, Nojun KwakCross-Modal Representation Learning Sparse Mixture-of-Experts
Seoul National University, Seoul, South Korea.
Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs
May 4, 2026Xin Zhang, Qiqi Tao, Jiawei Du +2Inference-Time Optimization Test-Time Optimization
Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Singapore University of Technology and Design +1
HY-Himmel Technical Report: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding
May 4, 2026Haopeng Jin, Hongzhu Yi, Wenlong Zhao +6Video Understanding Video-Language Models
Tencent · University of Chinese Academy of Sciences · Beijing Forestry University
VIDA: A Dataset for Visually Dependent Ambiguity in Multimodal Machine Translation
May 3, 2026Jingheng Pan, Xintong Wang, Longyue Wang +3Large Vision-Language Models Multimodal Model Evaluation
Department of Informatics, Universität Hamburg · Alibaba Group
Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering
May 3, 2026Yun Xing, Hanyuan Liu, Jiahao Nie +1Multimodal Large Language Models Vision-Language Grounding
Nanyang Technological University
Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models
May 3, 2026Yuriel Ryan, Hei Man Ip, Adriel Kuek +2VLM Robustness Multimodal Hallucination
Singapore University of Technology and Design · DSO National Laboratories · Massachusetts Institute of Technology
OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder
May 2, 2026Detao Bai, Shimin Yao, Weixuan Chen +4Cross-Modal Learning Video Understanding
1Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology
Valley3: Scaling Omni Foundation Models for E-commerce
May 2, 2026Zeyu Chen, Guanghao Zhou, Min Yang +6Unified Multimodal Models Multimodal Pretraining
Valley Team, ByteDance Group
LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations
May 1, 2026Huangbiao Xu, Huanqi Wu, Xiao Ke +1Missing-Modality Learning Multimodal Reasoning
Fujian Provincial Key Laboratory of Networking Computing and Intelligent Information Processing, College of Computer and Data Science, Fuzhou University, Fuzhou 350108, China · Engineering Research Center of Big Data Intelligence, Ministry of Education, Fuzhou 350108, China · Wangxuan Institute of Computer Technology, Peking University, Beijing 100871, China
GaMMA: Towards Joint Global-Temporal Music Understanding in Large Multimodal Models
May 1, 2026Zuyao You, Zhesong Yu, Mingyu Liu +3Multimodal Pretraining Multimodal Large Language Models
1Fudan University · 2ByteDance
Group Cognition Learning: Making Everything Better Through Governed Two-Stage Agents Collaboration
May 1, 2026Chunlei Meng, Pengbin Feng, Rong Fu +7Cross-Modal Learning Multi-Agent Collaboration
The College of Intelligent Robotics and Advanced Manufacturing, Fudan University · University of Southern California · University of Macau +5
Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL
Apr 30, 2026Sudong Wang, Weiquan Huang, Xiaomin Yu +9Multimodal Large Language Models Multimodal Reasoning
Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University · Tsinghua University +3
MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
Apr 30, 2026Junbo Cui, Bokai Xu, Chongyi Wang +33Efficient Multimodal Inference Unified Multimodal Models
MiniCPM-o Team, OpenBMB
COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts
Apr 30, 2026Bingli Wang, Huanze Tang, Haijun Lv +5Vision-Language Models Vision-Language Grounding
1Southeast University · 2Shanghai AI Laboratory
GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
Apr 29, 2026GLM-V Team, :, Wenyi Hong +95Multimodal Large Language Models Multimodal Reasoning
Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning
Apr 29, 2026Junwon You, Mihyun Jang, Sangwoo Mo +1Cross-Modal Learning Cross-Modal Representation Learning
1KAIST · 2POSTECH
M $^3$ -VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
Apr 28, 2026Jiatong Ma, Longteng Guo, Yuchen Liu +4Multimodal RAG Visual Question Answering
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models
Apr 27, 2026Weixing Wang, Liudvikas Zekas, Anton Hackl +5Unified Multimodal Models Cross-Modal Alignment
Hasso Plattner Institute / University of Potsdam, Potsdam, Germany
Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence
Apr 27, 2026NVIDIA, :, Amala Sanjay Deshmukh +216Efficient Multimodal Inference Multimodal Large Language Models
Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation
Apr 27, 2026Zhiheng Liu, Weiming Ren, Xiaoke Huang +12Unified Multimodal Models Multimodal Pretraining
Meta AI · University of Waterloo · The University of Hong Kong
X2SAM: Any Segmentation in Images and Videos
Apr 27, 2026Hao Wang, Limeng Qiao, Chi Zhang +4Image Segmentation Video Object Segmentation
Sun Yat-Sen University · 2 Peng Cheng Laboratory · 3 Meituan Inc
Omni-o3: Deep Nested Omnimodal Deduction for Deliberative Audio-Visual Reasoning
Apr 27, 2026Zhicheng Zhang, Wentao Gu, Weicheng Wang +5Audio-Visual Understanding Tree Search
Nankai University · Kuaishou Technology · Pengcheng Laboratory +1
Multimodal QUD: Inquisitive Questions from Scientific Figures
Apr 26, 2026Yating Wu, William Rudman, Venkata S Govindarajan +2Multimodal Grounding Scientific QA
The University of Texas at Austin · Ithaca College · UC Berkeley, BespokeLabs.ai
PushupBench: Your VLM is not good at counting pushups
Apr 25, 2026Shengzhi Li, Jiarun Chen, Karun Sharma +2VLM Evaluation Video-Language Model Evaluation
EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs
Apr 25, 2026He Hu, Tengjin Weng, Zebang Cheng +5Multimodal Large Language Models Multimodal Model Evaluation
Shenzhen University Shenzhen, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) Shenzhen, China · Queen Mary University of London +3

May 6, 2026, cs.CLJ/K move · Enter open · S save

HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities

Esra Dönmez, Pascal Tilli, Hsiu-Yu Yang, Thang Vu+1

Institute for Natural Language Processing, University of Stuttgart

Cross-Modal Learning Image Matching Hard Negative Mining Text-Image Alignment Vision-Language Alignment Vision-Language Pretraining+1

Open paper AskPDF arXiv

Image-Text-Matching (ITM) is one of the defacto methods of learning generalized representations from a large corpus in Vision and Language (VL). However, due to the weak association between the web-collected image-text pairs, models fail to show a fine-grained understanding of the combined semantics of these modalities. To address this issue we propose Hard Negative Captions (HNC): an automatically created dataset containing foiled hard negative captions for ITM training towards achieving fine-grained cross-modal comprehension in VL. Additionally, we provide a challenging manually-created test set for benchmarking models on a fine-grained cross-modal mismatch task with varying levels of compositional complexity. Our results show the effectiveness of training on HNC by improving the models' zero-shot capabilities in detecting mismatches on diagnostic tasks and performing robustly under noisy visual input scenarios. Also, we demonstrate that HNC models yield a comparable or better initialization for fine-tuning

Momentum

Related topics

Latest papers 413