Multimodal Instruction Tuning

Follow this topic Ask about this topic

Latest papers 80

7 papers in the last four weeks, down 36% on the four weeks before. 0.0% of all new papers.

All topics

System-Prompt Conditioning and Hidden-State Geometry in Four Open-Weight Models: Corrections and What Survives
Jul 10, 2026Jorge Castillo Sepúlveda, Marco Torres Yévenes, Juan Carlos LanasTransformer Interpretability Representation Geometry in Language Models
Axis Dynamics SpA, Santiago, Chile
NAVER LABS System Re-implementation for the IWSLT 2026 Instruction-Following Task
Jul 6, 2026Anand Kamble, Aniket TatheSpeech Language Models Multimodal Instruction Tuning
Florida State University · University of Illinois Urbana-Champaign
StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning
Jul 1, 2026Yuan Qing, Chengzhi Mao, Boqing GongVLM Evaluation VLM Robustness
Boston University, Boston, USA · Rutgers University, New Brunswick, USA
Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs
Jun 22, 2026Chuangxin Zhao, Canran Xiao, Siyuan Ma +5Visual Question Answering Cross-Modal Attention
The Hong Kong University of Science and Technology (Guangzhou) · Sun Yat-sen University · Nanyang Technological University +2
Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integration
Jun 15, 2026Xiwen Wei, Mark Nutter, Madhusudhanan Srinivasan +1Unified Multimodal Models Modality Imbalance
The University of Texas at Austin · Advanced Micro Devices, Inc.
ReasonRec: A Reasoning-Augmented Multimodal Agent for Unified Recommendation
Jun 8, 2026Yihua Zhang, Mingfu Liang, Jiyan Yang +11Efficient Multimodal Inference Multimodal CoT Reasoning
Meta AI · Michigan State University · The University of North Carolina at Chapel Hill +1
OmniCap-IF: Benchmarking and Improving Instruction Following Abilities for Omni-Video Captioning
Jun 7, 2026Jiahao Wang, An Ping, Yanghai Wang +13Video Captioning Multimodal Generation
NJU-LINK Team, Nanjing University · Kling Team, Kuaishou Technology
VCIFBench: Evaluating Complex Instruction Following for Video Understanding
Jun 3, 2026Huangchen Xu, Yuan Wu, Yi ChangVLM Evaluation Video Understanding
School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University · 3International Center of Future Science, Jilin University
Visual Instruction Tuning Aligns Modalities through Abstraction
Jun 2, 2026Luis Palacios, Lorenzo Basile, Diego Doimo +1VLM Adaptation VLM Interpretability
Area Science Park, Trieste, Italy
ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning
Jun 1, 2026Yu-Cheng Shi, Zhen-Hao Xie, Jun-Tao Tang +1Large Vision-Language Models Adapter Tuning
School of Artificial Intelligence, Nanjing University, China · State Key Laboratory of Novel Software Technology, Nanjing University, China
CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
Jun 1, 2026Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi +1Mixture-of-Experts Models Multimodal Instruction Tuning
School of Artificial Intelligence, Nanjing University, China · State Key Laboratory of Novel Software Technology, Nanjing University, China
Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging
Jun 1, 2026Minsik Choi, Geewook KimCommunication-Efficient Distributed Training Gradient Interference
NAVER Cloud AI · Korea University · KAIST AI
OphIn-500K: Curating Web-Scale Visual Instructions for Scaling Ophthalmic Multimodal Large Language Models
May 27, 2026Xuanzhao Dong, Wenhui Zhu, Xiwen Chen +13Medical Image Analysis Medical VQA
1Arizona State University · 2Clemson University · 3Washington University in St. Louis +5
OnceSelect: Reusable Data Selection for Efficient Multimodal Instruction Tuning
May 26, 2026Mingkang Dong, Muxin Pu, Hongyi Cai +5Instruction-Tuning Data Selection Multimodal Large Language Models
Faculty of Computer Science Universiti Malaya Kuala Lumpur, Malaysia · School of Information Technology Monash University Malaysia Subang Jaya, Selangor, Malaysia · Faculty of Computer Science University of Science and Technology Beijing Beijing, China
MULTISEISMO: A Multimodal Seismic Dataset and Model for Cross-Modal Seismic Understanding
May 25, 2026Sai Munikoti, Ian Stewart, Chengping Chai +4Multimodal Reasoning Seismology
Pacific Northwest National Laboratory (PNNL), Richland, WA 99354 USA · Oak Ridge National Laboratory (ORNL), Oak Ridge, TN 37830 USA · Sandia National Laboratory (SNL), Albuquerque, NM 87123 USA +1
Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning
May 25, 2026Jun-Tao Tang, Yu-Cheng Shi, Zhen-Hao Xie +1Continual Learning for LLMs Multimodal Large Language Models
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
MAGIC: Multimodal Alignment & Grounding-aware Instruction Coreset for Vision-Language Models
May 25, 2026Shristi Das Biswas, Kaushik RoyVision-Language Models Coreset Selection
Purdue University
Uni-Edit: Intelligent Editing Is A General Task For Unified Model Tuning
May 20, 2026Dian Zheng, Manyuan Zhang, Hongyu Li +4Text-Guided Image Editing Unified Multimodal Models
CUHK MMLab · Meituan · TJU +1
Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
May 13, 2026Qinwu Xu, Yifan Jiang, Haoyu RenMultimodal Grounding CoT Reasoning
Meta AI · UT Austin · Current independent researcher
Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents
May 11, 2026Shijue Huang, Hangyu Guo, Guanting Dong +8Multimodal IR Tool-Using Vision-Language Agents
Hong Kong University of Science and Technology · Renmin University of China · The Chinese University of Hong Kong +3
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
May 11, 2026Tao Hu, Da-Wei ZhouCross-Modal Learning Continual Learning for LLMs
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University
Efficient Data Selection for Multimodal Models via Incremental Optimization Utility
May 8, 2026Jinhao Jing, Qiannian Zhao, Chao Huang +1Data Valuation Multimodal Learning
StrLoRA: Towards Streaming Continual Visual Instruction Tuning for MLLMs
May 8, 2026Chang Che, Ziqi Wang, Hui Ma +2VLM Adaptation Low-Rank Adaptation
Hefei University of Technology · Tsinghua University
JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructions
May 6, 2026Leying Zhang, Bowen Shi, Haibin Wu +2Audio-Language Model Evaluation Zero-Shot Learning
Auditory Cognition and Computational Acoustics Lab, School of Computer Science& MoE Key Laboratory of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, 200240 P. R. China · independent researchers, United States
Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models
May 3, 2026Yuriel Ryan, Hei Man Ip, Adriel Kuek +2VLM Robustness Multimodal Hallucination
Singapore University of Technology and Design · DSO National Laboratories · Massachusetts Institute of Technology
Towards Customized Multimodal Role-Play
May 1, 2026Chao Tang, Jianzong Wu, Qingyu Shi +5Cross-Modal Learning Cross-Modal Alignment
1Peking University · 2Alibaba Group · 3Zhejiang University
OceanPile: A Large-Scale Multimodal Ocean Corpus for Foundation Models
Apr 25, 2026Yida Xue, Ningyu Zhang, Tingwei Wu +5AI for Science Multimodal Pretraining
College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China. · School of Software Technology, Zhejiang University, Ningbo 315048, China. · State Key Laboratory of Ocean Sensing, Hangzhou 311200, China. +2
Evian: Towards Explainable Visual Instruction-tuning Data Auditing
Apr 22, 2026Zimu Jia, Mingjie Xu, Andrew Estornell +1Large Vision-Language Models Multimodal Instruction Tuning
1The Hong Kong University of Science and Technology (Guangzhou) · 2ByteDance Seed
Medical Image Understanding Improves Survival Prediction via Visual Instruction Tuning
Apr 20, 2026Xixi Liu, Jorge Lazo, Andreas Hallqvist +8Clinical Outcome Prediction Medical VLMs
Chalmers University of Technology, Gothenburg, Sweden
AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers
Apr 17, 2026Edson Araujo, Saurabhchand Bhati, M. Jehanzeb Mirza +5Cross-Modal Learning Audio-Visual Understanding
University of Tübingen, Germany · MIT, Cambridge MA, USA · IBM Research, USA +2

Jul 10, 2026J/K move · Enter open · S save

System-Prompt Conditioning and Hidden-State Geometry in Four Open-Weight Models: Corrections and What Survives

Jorge Castillo Sepúlveda, Marco Torres Yévenes, Juan Carlos Lanas

Axis Dynamics SpA, Santiago, Chile

Transformer Interpretability Representation Geometry in Language Models Multimodal Instruction Tuning

Open paper AskPDF arXiv

Versions 1 and 2 of this preprint reported that an identity-specifying system prompt leaves a geometric fingerprint in the final-layer hidden-state trajectories of four open-weight language models, and that instruction tuning moves this fingerprint from the direction to the magnitude of the hidden-state vector. An audit of their code and data found the following. The curvature statistic described as Ollivier-Ricci curvature on Euclidean k-NN graphs was a non-standard Forman-type edge statistic on graphs built from temporal and cosine k-NN edges. Its released test permuted pooled edges instead of trajectories, and the published p-values came from unreleased code. The quantity reported as the norm of the first generated state is the state at the last prompt position, from which the first output token is predicted. The generic control prompt was matched to the identity prompt in characters, not in tokens. This version corrects the methods, withdraws the regime-specific claims (one model per regime) and the direction-to-magnitude claim, and re-analyzes the data with added controls. What survives is narrower. Centroid distance, maximum mean discrepancy and a linear probe separate every pair of prompt conditions in every model, while the curvature statistic exceeds its split-half noise floor in only four of twelve comparisons. In Gemma-4-E4B-it this state has a lower norm under the identity prompt than under a token-length-matched generic prompt (138.1 vs. 216.5; Cohen's d = -5.45; n = 20). Its direction also separates the conditions, and the effect fits the state's role in planning the output: the identity prompt instructs a pause before every answer, and the model opens 98 of 100 responses with a pause marker. When the first token is fixed, the norm ordering reverses. The base model continues the prompt template instead of answering. A redesigned follow-up study is in preparation.