Multimodal Instruction Tuning

Follow this topic Ask about this topic

Latest papers 79

7 papers in the last four weeks, down 36% on the four weeks before. 0.0% of all new papers.

All topics

CLIVO: Differentiable Cluster-Based Instruction Mixing with Learned Validation Objectives
Oct 12, 2026Yuhan Wang, Rui Hu, Batian Di +5Instruction-Tuning Data Selection Data Selection
Peking University · Renmin University of China · Zhejiang University +2
Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning
Oct 9, 2026Xi Zhang, Boyu Diao, Zesen Chen +3Continual Learning for VLMs Multimodal Instruction Tuning
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · ShanghaiTech University
Instruction-Conditioned Electromagnetic Spectrum Understanding via Budget-Adaptive Signal Tokenization
Oct 8, 2026Lei Zhai, Zhihao Chang, Shuyuan Yang +1Multimodal Token Compression Multimodal Instruction Tuning
School of Artificial Intelligence, Xidian University, Xi’an, China
CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing
Oct 7, 2026William Chen, Prem Seetharaman, Ke Chen +10Instruction-Based Video Editing Unified Multimodal Models
Adobe Research · Carnegie Mellon University
MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning
Oct 1, 2026Juekai Lin, Honglin Lin, Yuqian Yuan +7Reinforcement Learning Multimodal Reasoning
Zhejiang University · Shanghai Artificial Intelligence Laboratory, OpenDataLab · Shanghai Jiao Tong University +1
ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning
Sep 29, 2026Tao Hu, Zhinuo Zhou, Xialiang Tong +2Continual Learning Continual Learning for VLMs
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University · Noah’s Ark Lab, Huawei Technologies
ImIR: Image-Instruction Tuning for All-in-One Image Restoration
Sep 21, 2026Süleyman Aslan, Görkay Aydemir, Mısra Yavuz +5All-in-One Image Restoration Image Restoration
Codeway AI Research
AgriScope: Pixel-Grounded Multimodal Understanding for Agricultural Images
Sep 17, 2026Abderrahmene Boudiaf, Mohamad Alanssari, Irfan Hussain +1Agricultural Image Analysis Multimodal Grounding
Department of Computer Science, Khalifa University of Science and Technology, P.O. Box 127788, Abu Dhabi, United Arab Emirates · Department of Mechanical and Nuclear Engineering, Khalifa University of Science and Technology, P.O. Box 127788, Abu Dhabi, United Arab Emirates
Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillation
Sep 13, 2026Yirong Zeng, Zhang Sai, Yuxian Wang +4Multimodal Instruction Tuning RL for VLMs
Harbin Institute of Technology, SCIR Lab · Peking University
Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning
Sep 12, 2026Kunlun Xu, Yanqin Zhang, Wenwen Qiang +1Multimodal Instruction Tuning Multimodal Large Language Models
Wangxuan Institute of Computer Technology, Peking University, Beijing, China · Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
MM-IFEval-Pro: A Multilingual and Attack-Resistant Benchmark for Instruction-Following in Vision-Language Models
Sep 4, 2026Changming Xiao, Zhenliang Ni, Jinhui He +2VLM Robustness Adversarial Attacks on VLMs
Huawei Technologies Ltd. · Beijing, China · Shenzhen, China +1
From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding
Sep 1, 2026Raul Ortega, José Manuel Gómez-PérezVisual Question Answering Scientific QA
Language Technology Research Laboratory Expert.ai 17 Henri Dunant, 28036 Madrid, Spain
GUIDE: Guiding Internal Evidence with Language Instructions
Aug 31, 2026Soyeon Caren Han, Hyunsuk Chung, Jinwoo Kim +2Multimodal Robustness Multimodal Large Language Models
School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia · Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea · Brain Science Institute, Korea Institute of Science and Technology, Seoul, Republic of Korea +1
DICS: Exploring Data Intrinsic Consistency for Visual Instruction Selection
Aug 31, 2026Yuyang Hong, Jinhui Guo, Jiaqi Gu +6Instruction-Tuning Data Selection Training Data Selection
School of Artificial Intelligence, University of Chinese Academy of Sciences · 2MAIS, Institute of Automation, Chinese Academy of Sciences · 3Alibaba Token Hub, Alibaba Group
CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning
Aug 28, 2026Runze Liu, Naibin Gu, Mingxu Ai +4Continual Learning for VLMs Multimodal Instruction Tuning
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
BananaVLM: A Domain-Adapted Vision Language Model for Banana Crop Disease Diagnosis
Aug 27, 2026Sangam Kumar Jena, Pandarasamy ArjunanPlant Disease Classification VLM Adaptation
Department of Cyber-Physical Systems Indian Institute of Science Bangalore 560 012, India
LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning
Aug 27, 2026Muyao Yuan, Muyan Jiao, Jiangyong Ying +5Multimodal Instruction Tuning Cross-Modal Alignment
MOEKLINNS, and School of Computer Science and Technology Xi’an Jiaotong University Xi’an, China · E-surfing Vision Technology Co., Ltd China Telecom Hangzhou, China · China Telecom Xi’an, China
VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following
Aug 26, 2026Min Zeng, Guanxin Tan, Libin Cen +5Multimodal Instruction Tuning
vivo AI Lab
SciMIF: Understanding Multimodal Instruction Following in Scientific Domains
Aug 26, 2026Ye Shen, Yuting Zheng, Dun Pei +4Multimodal Model Evaluation Multimodal Large Language Models
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University
DistMoE: Private-data Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning
Aug 10, 2026Mainak Singha, Niccolò Biondi, Elisa Ricci +1Mixture-of-Experts Language Models VLM Adaptation
University of Trento, Italy · Fondazione Bruno Kessler, Italy · University of Bergamo, Italy
ToolVision: Learning When and How to Use Visual Tools with Capability-Aligned Supervision
Aug 9, 2026Delin Mao, Chenghao Sun, Jingwei Song +2Tool Use in VLMs Tool-Using Vision-Language Agents
Shanghai Jiao Tong University · University of Chinese Academy of Sciences · Fudan University
NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs
Aug 8, 2026Jiayue Jin, Jingwei Zhang, Chen Wang +2Multimodal Large Language Models Multimodal Instruction Tuning
College of Intelligent Robotics and Advanced Manufacturing, Fudan University · Zhongguancun Academy · Tianjin University +3
PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesis
Aug 5, 2026Xiaomin He, Dongling Xiao, Jiahao Xie +4Multimodal Generation Rubric-Based Evaluation
ByteDance · Peking University
Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning
Aug 2, 2026Huiyu Yi, Yongqi Xu, Bogang Zhang +5Class-Incremental Learning Adaptive Model Routing
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China
Progressive Multimodal Alignment for Continual Instruction Tuning
Jul 29, 2026Duzhen Zhang, Yahan Yu, Qiaoyi Su +2Cross-Modal Alignment Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences · Kyoto University +2
Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution
Jul 29, 2026Mingkuan Feng, Zhengqi Wen, Jianhua TaoVLM Adaptation Large Vision-Language Models
Tsinghua University · Beijing National Research Center for Information Science and Technology
ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships
Jul 16, 2026Xinyu Liu, Shihao Li, Weihong Lin +10Multimodal Grounding Video Diffusion Models
HKUST · Kling Team · NJU +3
Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncology
Jul 16, 2026Sinyoung Ra, Jonghun Kim, Hyunjin ParkRadiology Report Generation 3D Medical Imaging
Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Republic of Korea
UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp
Jul 12, 2026Xiyu Wei, Qingwei Zong, Zhuocheng Yu +1Multimodal QA AI Agent Benchmarks
Key Laboratory of Computational Linguistics, MOE, Peking University · School of Software and Microelectronics, Peking University · School of Computer Science, Peking University
System-Prompt Conditioning and Hidden-State Geometry in Four Open-Weight Models: Corrections and What Survives
Jul 10, 2026Jorge Castillo Sepúlveda, Marco Torres Yévenes, Juan Carlos LanasTransformer Interpretability Representation Geometry in Language Models
Axis Dynamics SpA, Santiago, Chile

Oct 12, 2026J/K move · Enter open · S save

CLIVO: Differentiable Cluster-Based Instruction Mixing with Learned Validation Objectives

Yuhan Wang, Rui Hu, Batian Di, Tan Jiang+4

Peking University · Renmin University of China · Zhejiang University+2

Instruction-Tuning Data Selection Data Selection Data Mixture Optimization Multimodal Instruction Tuning

Open paper AskPDF arXiv

Target-aware multimodal instruction selection requires useful examples and an appropriate balance of supervision. Sample-level rankings can overrepresent similar content without optimizing this balance; group-level methods face coarse units, predefined set-level validation signals, and costly or inaccurate mixture search. We propose CLIVO, a target-aware framework connecting fine-grained semantic allocation, automatically learned validation objectives, and differentiable mixture search. Specifically, semantic clusters provide independently adjustable shares at the hundred-cluster scale. Correlations between per-example merged-proxy losses and real probe performance identify target-relevant validation examples; learned nonnegative weights combine their losses into an objective. A single backward pass through the merged proxy supplies gradients for all shares, avoiding an additional mixture-response regression model. Selecting 100K instructions from an 8M pool, CLIVO attains 97.2% of the full-data four-target average score with only 1.25% of the data. It improves on the strongest evaluated selection baseline by 1.16 points using random within-cluster sampling; combining learned quotas with sample ranking increases this advantage to 1.55 points. Analyses show that finer units benefit most targets, short-trained experts preserve mixture responses, sample weighting retains useful validation information, and direct search consistently reduces the learned objective. CLIVO also supports interpretable allocation, modular selection, and reuse across targets and granularities.