Multimodal Large Language Models

Also known as MLLM

Follow this topic Ask about this topic

Momentum

67 papers in the last four weeks, up 91% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 684

All topics

OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs
May 26, 2026Guangzhi Sun, Yixuan Li, Yudong Yang +1Streaming Video Understanding Multimodal Large Language Models
ByteDance · Department of Engineering, University of Cambridge · Tsinghua University
Hallucination Behavior in Multimodal LLMs Across Agricultural Image Interpretation and Generation Tasks
May 26, 2026Partho Ghose, Al Bashir, Prem Raj +1Agricultural Image Analysis VLM Hallucination
Department of Biological & Agricultural Engineering, Texas A&M AgriLife Research, Texas A&M University System, Dallas, TX 75252, USA.
Touch-R1: Reinforcing Touch Reasoning in MLLMs
May 26, 2026Yingxin Lai, Yafei Zhou, Fucai Zhu +2Tactile Sensing Multimodal Large Language Models
1Xiamen University · 5Daimon Robotics · 2Great Bay University +2
Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning
May 26, 2026Junkai Chen, Yuhao He, Junxiang You +3VLM Unlearning Multimodal Large Language Models
Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, UCAS
LiveK12Bench: Have Large Multimodal Models Truly Conquered High School-level Examinations?
May 26, 2026Xiaohan Wang, Mingze Yin, Yilin Zhao +2Mathematical Reasoning Benchmarks Educational Assessment
Tencent PCG · College of Computer Science and Technology, Zhejiang University
OnceSelect: Reusable Data Selection for Efficient Multimodal Instruction Tuning
May 26, 2026Mingkang Dong, Muxin Pu, Hongyi Cai +5Instruction-Tuning Data Selection Multimodal Large Language Models
Faculty of Computer Science Universiti Malaya Kuala Lumpur, Malaysia · School of Information Technology Monash University Malaysia Subang Jaya, Selangor, Malaysia · Faculty of Computer Science University of Science and Technology Beijing Beijing, China
DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding
May 26, 2026Peng Zhang, Guanghao Zhang, Wanggui He +10Multimodal Large Language Models Video Reasoning
Zhejiang University · Alibaba Group
Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
May 25, 2026Bonan Ding, Umair Nawaz, Ufaq Khan +5Cross-Modal Learning Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence · Tianjin University · Chongqing University +1
Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning
May 25, 2026Jun-Tao Tang, Yu-Cheng Shi, Zhen-Hao Xie +1Continual Learning for LLMs Multimodal Large Language Models
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Models
May 25, 2026Yulin Yuan, Hongshuo Zhao, Xiangming MengLanguage Model Decoding Multimodal Large Language Models
Zhejiang University · ZJUI-UIUC Institute
AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution
May 25, 2026Zehao Wang, Yihan Zeng, Zidong Gong +5RL for Language Model Reasoning Multimodal Large Language Models
Harbin Institute of Technology · Huawei Noah’s Ark Lab · Independent Researcher
StructBreak: Structural Cognitive Overload-Induced Safety Failures in MLLMs
May 25, 2026Yang Luo, Xinran Liu, Tiantian Ji +3Adversarial Attacks on VLMs Language Model Safety Evaluation
Key Laboratory of Trustworthy Distributed Computing and Service (MoE), Beijing University of Posts and Telecommunications · Institute of Computing Technology, Chinese Academy of Sciences
MetaphorVU: Towards Metaphorical Video Understanding
May 25, 2026Zhuoqun Li, Boxi Cao, Guiping Jiang +13VLM Evaluation Figurative Language Understanding
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Kuaishou Technology +1
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
May 25, 2026Zili Zhang, Chengxu Yang, Shenglong Zhang +8Deep Learning Optimization Multimodal Large Language Models
1Peking University · †Work done during internships at Xiaohongshu, Inc. · 2Independent Researcher +2
Dual-Pathway Geometry-Aware MLLM for Spatial Intelligence
May 25, 2026Yufei Zheng, Xuhan Zhu, Zide Liu +9Visual Spatial Reasoning Metric Depth Estimation
University of Science and Technology of China · Li Auto Inc. · Shanghai Jiao Tong University
Divide-and-Conquer Inference for Large-Scale Visual Recognition with Multimodal Large Language Models
May 24, 2026Zhipeng Ye, Jiaqi Huang, Feng Jiang +5Efficient Multimodal Inference Multimodal Large Language Models
Taizhou Institute of Science and Technology, Nanjing University of Science and Technology, Taizhou, 225300, Jiangsu, China · Department of Intelligence Science, Xi’an Jiaotong-Liverpool University, Suzhou, 215123, Jiangsu, China · Department of Statistical Sciences, University of Toronto, Toronto, M5S 1A1, Ontario, Canada +1
EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy
May 23, 2026Jinzhao Li, Yinuo Chen, Dongxu Piao +9Spatial Reasoning Benchmarks LLM Evaluation
College of AI, Tsinghua University · 2ByteDance · 3State Key Laboratory of General Artificial Intelligence, BIGAI
Beyond Binary Edits Robust Multimodal Knowledge Editing with Adversarial Subspace Alignment
May 22, 2026Haoyuan Wang, Xiaohao Liu, Jiajie Su +2Multimodal Robustness Adversarial Robustness
Zhejiang University · National University of Singapore · Jiangxi Normal University
ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models
May 22, 2026Fen Wang, Zekai Shao, Qiman Kang +5Data Visualization Multimodal Large Language Models
School of Data Science, Fudan University · Zhengzhou Zhongke Institute of Integrated Circuit and System Application · School of Computer Science, Fudan University
From Recognition to Reasoning: Benchmarking and Enhancing MLLMs on Real-World Receipt Document Understanding
May 21, 2026Yandi Wang, Libin Zhan, Ziwei Huang +5Document Understanding Document Information Extraction
Zhejiang University, China
Enhancing Multimodal Large Language Models for Safety-Critical Driving Video Analysis
May 21, 2026Tomaso Trinci, Henrique Piñeiro Monteagudo, Leonardo TaccariMultimodal Large Language Models Autonomous Driving
Verizon Connect, Florence, Italy.
Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?
May 21, 2026Luca Modica, Filip Landin, Mehrdad Farahani +3Multimodal Large Language Models Factual Knowledge in Language Models
Zenseact · Chalmers University of Technology · University of Gothenburg +2
Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention
May 21, 2026Changyuan Tian, Zhicong Lu, Huaxing Liu +7Multimodal Large Language Models Multimodal Reasoning
AMAP, Alibaba Group · University of Chinese Academy of Sciences · Tsinghua University +1
LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
May 21, 2026Yifan Dai, Zhenhua Wu, Bohan Zeng +18Cross-Modal Learning Audio-Visual Understanding
School of AI, Shanghai Jiao Tong University · 2Kling Team, Kuaishou Technology · 3Peking University +5
MLLMs Know When Before Speaking: Revealing and Recovering Temporal Grounding via Attention Cues
May 21, 2026Dazhao Du, Liao Duan, Jian Liu +5Temporal Video Grounding Multimodal Large Language Models
Hong Kong University of Science and Technology · Tencent · Xi’an Jiaotong University
Seizure-Semiology-Suite (S3): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding
May 21, 2026Lina Zhang, Tonmoy Monsoor, Peizheng Li +23Video Understanding Multimodal Large Language Models
University of California, Los Angeles · Mercedes-Benz AG · University of Pittsburgh +4
Multimodal LLMs under Pairwise Modalities
May 20, 2026Yan Li, Yunlong Deng, Yuewen Sun +3Cross-Modal Learning Representation Identifiability
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University
Frequency-Domain Regularized Adversarial Alignment for Transferable Attacks against Closed-Source MLLMs
May 20, 2026Leitao Yuan, Qinghua Mao, Daizong Liu +5Adversarial Attacks on VLMs Multimodal Large Language Models
Shanghai Artificial Intelligence Laboratory · Zhejiang University · Shanghai Jiao Tong University +3
Miller-Index-Based Latent Crystallographic Fracture Plane Reasoning and generation with Vision-Language Models
May 19, 2026Qinwu Xu, Xiaofu Ma, Yifan JiangMultimodal Large Language Models Multimodal Reasoning
Independent research
Stage-adaptive Token Selection for Efficient Omni-modal LLMs
May 19, 2026Zijie Xin, Jie Yang, Ruixiang Zhao +4Efficient Multimodal Inference Multimodal Large Language Models
Renmin University of China · WeChat Vision, Tencent Inc.

May 26, 2026, cs.AIJ/K move · Enter open · S save

OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs

Guangzhi Sun, Yixuan Li, Yudong Yang, Chao Zhang

ByteDance · Department of Engineering, University of Cambridge · Tsinghua University

Streaming Video Understanding Multimodal Large Language Models Multimodal Token Compression KV-Cache Compression Video Token Compression

Open paper AskPDF arXiv

Audio-visual large language models (LLMs) hold strong promise for long-form video understanding, yet their long-video inference is fundamentally limited by the linear growth of video tokens and key-value (KV) caches. We present OmniMem, a memory-efficient streaming framework designed specifically for audio-visual LLMs. Unlike existing compression methods that treat all tokens uniformly, OmniMem introduces a modality-aware memory allocation strategy that separately manages visual and audio contexts, addressing the severe token imbalance between the two modalities. OmniMem further preserves informative and non-redundant KV states through perturbation-aware memory selection, enabling compact memory without sacrificing long-range understanding. To strengthen compression under realistic deployment constraints, we also explore budget-aware fine-tuning, which encourages the model to consolidate useful information into retained memory. Experiments on VideoMME Long, LVBench, and LVOmniBench with video-SALMONN 2+ and Qwen-2.5-Omni show that OmniMem consistently improves over strong training-free compression baselines by 2-4% absolute accuracy under the same memory budgets, with an additional 1-2% gain after fine-tuning.

Momentum

Related topics

Latest papers 684