Multimodal Understanding

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, up 233% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 416

All topics

Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning
Jun 30, 2026Kaitao Chen, Weiqian Zhao, Jiamin Wu +6Efficient Multimodal Inference Medical VQA
1Fudan University · 2Shanghai Artificial Intelligence Laboratory · 3Shanghai Jiao Tong University +1
MS-Resampler: Multi-Scope Visual Resampling for Efficient Multimodal LLMs
Jun 30, 2026Zhongyang Li, Yaqian Li, Faming Fang +6Multimodal Large Language Models Multimodal Fusion
Li Auto Inc. · East China Normal University.
Learning to Deny: Action Denial in Multimodal Large Language Models
Jun 30, 2026Raiyaan Abdullah, Shehreen Azad, Yogesh Singh RawatMultimodal Large Language Models Causal Reasoning in Videos
Institute of Artificial Intelligence, University of Central Florida, USA
Orca: The World is in Your Mind
Jun 29, 2026Yihao Wang, Yuheng Ji, Mingyu Cao +54World Model Learning Multimodal Pretraining
Latent Noise Mask for Reducing Visual Redundancy in Multimodal Large Language Models
Jun 29, 2026Kai Jiang, Ruishu Zhu, Siqi Huang +2Visual Question Answering Multimodal Large Language Models
School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University · Institute of Artificial Intelligence, China Telecom (TeleAI) · 3Fudan University +1
Consistency as Inductive Bias: Learning Cross-View Invariance for Robust Multimodal Reasoning
Jun 29, 2026Xin Zou, Haolin Deng, Yibo Yan +6Multi-View Consistency Multimodal Robustness
HKUST (GZ) · HKUST · UIC +1
Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation
Jun 29, 2026Xin Zou, Haolin Deng, Yibo Yan +5Multimodal Large Language Models Multimodal Hallucination
HKUST (GZ) · HKUST · OPPO AI Center
Reliability-Prioritized Fine-Grained Generation in Multimodal Large
Jun 28, 2026Xiaomeng Fan, Wei Wu, Yuwei Wu +9Direct Preference Optimization Multimodal Large Language Models
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University · Department of Electrical and Computer System Engineering, Monash University
MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein
Jun 28, 2026Hong-Han Wang, Yuntao Wang, Hu DingLarge Vision-Language Models VLM Reasoning
University of Science and Technology of China, Hefei, China
Personalizing MLLMs via Reinforced Multimodal Reference Game
Jun 27, 2026Deepayan Das, Davide Talon, Yiming Wang +2Multimodal Large Language Models Personalized Language Models
University of Trento, Trento, Italy · Fondazione Bruno Kessler, Trento, Italy
COMPASS: Grounding Composition-Intent Guidance in Unified Multimodal Models
Jun 27, 2026Ziqi Zhou, Weize Quan, Mining Tan +6Multimodal Grounding Unified Multimodal Models
University of Edinburgh, United Kingdom · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China +1
Understanding How MLLMs Describe Artworks Using Token Activation Maps
Jun 26, 2026Nicola Fanelli, Pasquale De Marinis, Raffaele Scaringi +3VLM Interpretability Vision-Language Grounding
Department of Computer Science University of Bari Aldo Moro Bari, Italy
MER-R1: Multimodal Emotion Reasoning via Slow-Fast Thinking Synergy
Jun 26, 2026Zhiyuan Han, Beier Zhu, Wenwen Tong +8RL for Language Model Reasoning Multimodal Large Language Models
University of Science and Technology of China · 2SenseTime Research · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +1
Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards
Jun 25, 2026Ritesh Thawkar, Shravan Venkatraman, Omkar Thawakar +5Unified Multimodal Models Multimodal Generation
1Mohamed bin Zayed University of Artificial Intelligence · 4Linköping University · 3Australian National University +1
ViQ: Text-Aligned Visual Quantized Representations at Any Resolution
Jun 25, 2026Xumin Yu, Zuyan Liu, Zhenyu Yang +5Visual Representation Learning Multimodal Pretraining
Tencent HY Vision Team · Tsinghua University · Institute of Automation, CAS +1
See & Sniff: Learning Visuo-Olfactory Representations
Jun 25, 2026Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu +2Representation Learning Cross-Modal Representation Learning
Korea Advanced Institute of Science and Technology, Korea · Hankuk University of Foreign Studies, Korea · Ulsan National Institute of Science and Technology, Korea
Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation
Jun 25, 2026Jinyu Liu, Xincheng Shuai, Henghui Ding +1Unified Multimodal Models Multimodal Foundation Models
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China
Bridging Vision and Language Concepts through Optimal Transport Semantic Flow
Jun 25, 2026Chenyang Zhang, Anqi Dong, Guangming Zhu +4Concept Bottleneck Models VLM Interpretability
School of Computer Science and Technology, Xidian University, Xi’an, China · KTH Royal Institute of Technology, Stockholm, Sweden
SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Context
Jun 25, 2026Qinkai Zhang, Yanyan Zhao, Xin Lu +3LLM Personalization Multimodal Large Language Models
Harbin Institute of Technology
Position Rebinding Cache Reuse: Replay-Free Visual Revisiting for Interleaved Multimodal Reasoning
Jun 25, 2026Mengzhao Wang, Yanli Ji, Wangmeng Zuo +2Efficient Multimodal Inference KV Caching
Sun Yat-sen University · Shenzhen Loop Area Institute · Harbin Institute of Technology (HIT) +3
What We are Missing in Multimodal LLM Evaluation?
Jun 24, 2026Po-han Li, Shenghui Chen, Sandeep Chinchali +1Spatiotemporal Reasoning Multimodal Large Language Models
The University of Texas at Austin, USA
Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models
Jun 24, 2026Akshay Paruchuri, Sanmi Koyejo, Ehsan AdeliMultimodal Robustness LLM Auditing
Stanford University
TriViewBench: Controlled Complexity Scaling for Multi-View Structural Reasoning in MLLMs
Jun 24, 2026Yu-Yang Chen, Lan-Zhe GuoSpatial Reasoning Benchmarks Visual Reasoning
School of Intelligence Science and Technology, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China
From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
Jun 24, 2026Haoxiang Sun, Tao Wang, Li Yuan +2Vision-Language Models Multimodal Large Language Models
School of Computer Science, Sichuan University, No. 24 South Section 1, Yihuan Road, Chengdu, 610065, Sichuan, China · School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School, 2199 Lishui Road, Shenzhen, 518055, Guangdong, China · Institute of Artificial Intelligence (TeleAI), China Telecom and Northwestern Polytechnical University, 127 Youyi West Road, Xi’an, 710072, Shaanxi, China
SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessity
Jun 24, 2026Tianchen Guo, Chen Liu, Ling Chen +1Visual Question Answering Multi-View Consistency
The University of Queensland · Follow Me AI Pty LTD · University of Technology Sydney +1
Omni-Perception Policy Optimization for Multimodal Emotion Reasoning
Jun 24, 2026Zhiyuan Han, Beier Zhu, Wenwen Tong +6Multimodal Reasoning Multimodal Hallucination
University of Science and Technology of China · SenseTime · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +1
iFLYTEK-Embodied-Omni Technical Report
Jun 24, 2026Yuan Zhang, Jingfei Ni, Guanchen Lu +12Unified Multimodal Models Robot Motion Generation
1iFLYTEK · University of Science and Technology of China · 2LindenBot
ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answering
Jun 23, 2026Zhentao Guo, Chen Duan, Tongkun Guan +3Multimodal Large Language Models Video QA
Meituan · Shanghai Jiao Tong University
S1-Omni-Image: A Unified Model for Scientific Image Understanding, Generation, and Editing
Jun 23, 2026Qingxiao Li, Zikai Wang, Qingli Wang +1AI for Science Unified Multimodal Models
XScience Lab, Wenge AI
AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression
Jun 23, 2026Yijing Chen, Wenhui Tan, Xiaoyi Yu +7Audio-Visual Understanding Multimodal Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China · Huawei Inc.

Jun 30, 2026, cs.CVJ/K move · Enter open · S save

Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning

Kaitao Chen, Weiqian Zhao, Jiamin Wu, Qihao Zheng+5

1Fudan University · 2Shanghai Artificial Intelligence Laboratory · 3Shanghai Jiao Tong University+1

Efficient Multimodal Inference Medical VQA Multimodal Reasoning Medical VLMs Visual Token Pruning RL for VLMs+1

Open paper AskPDF arXiv

Vision-language models (VLMs) combining reinforcement learning (RL) ignite remarkable progress in multimodal reasoning, yet still struggle with medical images, which typically exhibit extremely sparse visual evidence to inform clinical decision-making. We recognize that pruning visual tokens outside the grounding region greatly enhances medical reasoning. However, a united RL framework for active visual token pruning (VTP) and medical multimodal reasoning remains unestablished. Here, we propose a dual-stream RL framework, ViToS, to fulfill token pruning and question answering. ViToS trains one policy model with two task branches, where one focuses on grounding while the other conducts token-sparse reasoning after VTP. Furthermore, we solve the coupled policy learning problem by introducing the cross-feedback sequential optimization, avoiding gradient conflict and facilitating convergence of the shared policy model. Evaluated on seven medical benchmarks, our method reduces visual tokens to 77% of the original sequence length while achieving a 108.27% relative performance on Lingshu-7B and 104.16% relative performance on HuatuoGPT-Vision-7B. Overall, ViToS delivers superior performance and inference speedup, establishing an efficient paradigm for medical multimodal reasoning.

Momentum

Related topics

Latest papers 416