Multimodal Large Language Models

Also known as MLLM

Follow this topic Ask about this topic

Momentum

67 papers in the last four weeks, up 91% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 676

All topics

UniData: Universal Multimodal Instruction Generation Pipeline
Oct 8, 2026Jiaqi Tang, Yi-Feng Wu, Yuting Zhang +12Synthetic Data Generation for Language Models Multimodal Generation
The Hong Kong University of Science and Technology · ATH, Alibaba Group · The Hong Kong University of Science and Technology (Guangzhou) +2
From Pixel to Coding: Evaluating the Figure Reproduction Capabilities of MLLMs
Oct 7, 2026Zijian Chen, Zhengyu Chen, Bohan Liang +7Multimodal Large Language Models Multimodal Model Evaluation
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University, Shanghai, 200240, China. · Shanghai Artificial Intelligence Laboratory, Shanghai, 200030, China. · Macao Polytechnic University, Macao, 999078, China.
TiTok: Audio-Visual LLM for Multi-Segment Temporal Grounding
Oct 7, 2026Eunji Shin, Dahyun Choi, Seungyeon Jo +2Multimodal Grounding Multimodal Large Language Models
Department of Artificial Intelligence, Ewha Womans University · Division of Artificial Intelligence & Software, Ewha Womans University
RSJEV: Discriminative Remote Sensing Scene Classification with Multimodal Large Language Models
Oct 6, 2026Dongchen Si, Di Wang, Mingzhen Xu +3Remote Sensing Image Understanding Efficient Multimodal Inference
School of Computer Science, Wuhan University, Wuhan 430072, China · State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China
DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models
Oct 6, 2026Shuo Yang, Changbai Li, Linlin Yang +6Multimodal Large Language Models Token Pruning
Beihang University · Communication University of China · National University of Singapore
VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs
Oct 6, 2026Yuan Feng, Qize Yang, Ruizhe Chen +9Efficient Multimodal Inference Multimodal Large Language Models
University of Science and Technology of China · Alibaba Token Hub, Alibaba Group · The Chinese University of Hong Kong +1
Two Vectors Replace In-Context Demos: Structured Task Adaptation via Embeddings
Oct 6, 2026Xi Ding, Naichen Shi, Jiawei ZhangMultimodal ICL Task Vectors
University of Wisconsin–Madison · Northwestern University
Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection
Oct 5, 2026Zhaolin Cai, Huiyu Duan, Liu Yang +5Multimodal Large Language Models Active Visual Perception
Shanghai Jiao Tong University · Xinjiang University · Beijing Jiaotong University +1
Human-Like Attention? A Psychophysical Comparison of Visual Search in Humans and MLLMs
Oct 4, 2026Renchi Zhang, Joost C. F. de Winter, Dimitra Dodou +2Multimodal Large Language Models Visual Search
Faculty of Mechanical Engineering, Delft University of Technology, Delft, The Netherlands
A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models
Oct 4, 2026Yilin Yang, Jun-Tao Tang, Kengyi Wang +3VLM Evaluation Multimodal Large Language Models
School of Artificial Intelligence, Shanghai Jiao Tong University · Nanjing University · Fudan University +1
OmniConfess: Eliciting Token Confessions to Mitigate Omni-Modal Hallucination
Oct 2, 2026Huiqiang Rong, Haoran Luo, Hui Feng +4Multimodal Large Language Models Multimodal Hallucination
Beijing University of Posts and Telecommunications · Nanyang Technological University, Singapore
OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning
Oct 1, 2026Haibo Wang, Jiteng Mu, Jialu Li +5Audio-Visual Understanding RL for Language Model Reasoning
University of California, Davis · Adobe Research
Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes
Oct 1, 2026Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc +2VLM Distillation Multimodal Large Language Models
Sorbonne Universit´e, CNRS, ISIR, F-75005 Paris, France · ILLS, CNRS, Montreal, QC H2S 3H1 · Institut universitaire de France (IUF)
MWOP: Modality-aware Width-wise Operation Pruning for Efficient MLLMs
Oct 1, 2026Xudong Wang, Hao Wu, Haozhe Hu +5LLM Pruning Efficient VLM Inference
EIT-NLP Lab, Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · The Hong Kong Polytechnic University +1
HAWK: Rethinking Multimodal Drafting for Speculative Decoding
Sep 30, 2026Wenhan Yang, Anirudh Rao, Ashwin ChandraEfficient Multimodal Inference VLM Distillation
University of California, Los Angeles · Samsung Research America
Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis
Sep 30, 2026Tian Xia, Minghao Liu, Yiqing Liang +2Medical Diagnosis Multimodal Large Language Models
Harvard University · University of California, Santa Cruz · Brown University +2
Learning to Reason with Compressed Context: Ground-Truth-Free Adaptation of OmniLLMs via Self-Distillation
Sep 30, 2026Jianghao Wang, Ke Meng, Jian Li +7Efficient Multimodal Inference Multimodal Large Language Models
School of Computer Science, Wuhan University · Didi Chuxing
Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
Sep 30, 2026Zhihan Zhang, Lizi LiaoToken-Level Credit Assignment Multimodal Large Language Models
Singapore Management University Singapore, Singapore
MM-FinEval: A Multi-Task Multimodal Benchmark for Real-World Financial Forecasting
Sep 29, 2026Dong Shu, Yanguang Liu, Huopu Zhang +4Financial Forecasting Multimodal Large Language Models
Northwestern University · New Jersey Institute of Technology · Georgia Institute of Technology +2
Exemplar2VQA: A Scalable Exemplar-Driven Visual Question Answering Generation Framework via Multi-Agent Coding
Sep 29, 2026Jiayu Ying, Qijian Tian, Ruijie Xu +4Visual Question Answering Multimodal Large Language Models
East China Normal University · Shanghai Jiao Tong University · Fudan University +1
Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models
Sep 29, 2026Yu Zhang, Pingrui Zhang, Xuefeng Bai +3LLM Hallucination Mitigation Hallucination in Language Models
Harbin Institute of Technology, Shenzhen, China
Why MLLMs Struggle to Count: Overcoming Individuation and Aggregation Bottlenecks with ConvStack
Sep 29, 2026Liwei Che, Yihao Quan, Sen Fang +4Object Counting Multimodal Large Language Models
Department of Computer Science, Rutgers University · Department of Computer Science, University of Washington
OmniRoute: Mapping Temporal Semantic Evidence to Audio-Visual Token Budgets for Efficient Omnimodal Large Language Models
Sep 29, 2026Yuchen Deng, Zidang Cai, Feidiao Yang +4Efficient Multimodal Inference Audio Token Compression
Shenzhen International Graduate School, Tsinghua University, China · Pengcheng Laboratory, China
Calibrate the Decisions That Change the Future: On-Policy Post-Training Quantization for Multimodal Large Language Models
Sep 29, 2026Wenxiao Fan, Jingling Fu, Lichen Ma +6LLM Quantization VLM Quantization
School of Computer Science and Technology, Beijing Institute of Technology · Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University
Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs
Sep 29, 2026Yueran Ma, Ronghao LinAudio QA Multimodal Large Language Models
The University of Queensland, Brisbane, Australia · Shenzhen University, Shenzhen, China
Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning
Sep 29, 2026Cheng Ye, Weidong Chen, Zhaobo Qi +2Multimodal Large Language Models Multimodal Emotion Recognition
University of Science and Technology of China, Hefei · Harbin Institute of Technology, Weihai
AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models
Sep 28, 2026Konstantinos D. Polyzos, Eleni Oikonomou, Tara JavidiMultimodal Large Language Models Kernel Methods
University of California San Diego
Render Before Reading: Visual Rendering as a Prompt Injection Defense
Sep 28, 2026Jie Zhang, Andrei Baroian, Jan N. van Rijn +2Multimodal Large Language Models Modality Gap in VLMs
ETH Zurich · Leiden University
How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
Sep 28, 2026Lin Chen, Bolin Ni, Qi Yang +6Multimodal Pretraining Multimodal Large Language Models
CASIA · UCAS · Foundation Model Department, Tencent
SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Models
Sep 28, 2026Tianxiang Chen, Zhentao Tan, Zi Ye +9Efficient Multimodal Inference Transformer
End-User Intelligent Computing BU, Alibaba Cloud · Alibaba Group · Department of Computer Science, Maynooth University +2

Oct 8, 2026, cs.AIJ/K move · Enter open · S save

UniData: Universal Multimodal Instruction Generation Pipeline

Jiaqi Tang, Yi-Feng Wu, Yuting Zhang, Hao Lu+11

The Hong Kong University of Science and Technology · ATH, Alibaba Group · The Hong Kong University of Science and Technology (Guangzhou)+2

Synthetic Data Generation for Language Models Multimodal Generation Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) are increasingly being applied in a wider range of real-world scenarios. However, due to the substantial labor cost, creating high-quality multimodal instruction datasets for MLLMs remains a significant challenge. Although some methods propose to generate instruction data, they often face limitations in modality support and struggle with generating multi-round instructions. To address these problems, we introduce UniData, a universal instruction generation pipeline, to transform simple user requirements into multi-round, multimodal instructions. Specifically, UniData first expands user requirements into multiple diverse events. Using these events, UniData then integrates an any-to-any large model for multimodal instruction generation. Finally, UniData enhances data quality by correcting irrelevant and redundant inference flow, leveraging correlations between instruction rounds. To train this pipeline, we also build UniDataset, a dataset comprising 20,000 entries across nine modalities for improved multimodal generation. Our experiments demonstrate that UniData achieves SOTA performance in data quality and can also enhance the understanding and generation capabilities of other multimodal models.

Momentum

Related topics

Latest papers 676