Visual Instruction Tuning

Latest papers

12 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Instruction Mining From Images: Constructing a Synthetic Dataset for Multimodal Learning ILSP DCVLR Competition Submission
Oct 14, 2026Eleni E. Kasoura, Dimitrios Damianos, Georgios Paraskevopoulos +1Visual Instruction Tuning Synthetic Data Generation for Language Models
Institute of Language and Speech Processing Athena Research Center Athens, GR 151 25
ImIR: Image-Instruction Tuning for All-in-One Image Restoration
Sep 21, 2026Süleyman Aslan, Görkay Aydemir, Mısra Yavuz +5Visual Instruction Tuning All-in-One Image Restoration
Codeway AI Research
From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding
Sep 1, 2026Raul Ortega, José Manuel Gómez-PérezVisual Question Answering Scientific QA
Language Technology Research Laboratory Expert.ai 17 Henri Dunant, 28036 Madrid, Spain
Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncology
Jul 16, 2026Sinyoung Ra, Jonghun Kim, Hyunjin ParkRadiology Report Generation 3D Medical Imaging
Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Republic of Korea
Vision as Unified Multimodal Generation
Jul 7, 2026Xiaoyang Han, Jianhua Li, Kewang Deng +14Unified Multimodal Models Multimodal Generation
SenseTime Research · Peking University · Shanghai Jiao Tong University +3
Visual Instruction Tuning Aligns Modalities through Abstraction
Jun 2, 2026Luis Palacios, Lorenzo Basile, Diego Doimo +1VLM Adaptation VLM Interpretability
Area Science Park, Trieste, Italy
StrLoRA: Towards Streaming Continual Visual Instruction Tuning for MLLMs
May 8, 2026Chang Che, Ziqi Wang, Hui Ma +2VLM Adaptation Low-Rank Adaptation
Hefei University of Technology · Tsinghua University
Medical Image Understanding Improves Survival Prediction via Visual Instruction Tuning
Apr 20, 2026Xixi Liu, Jorge Lazo, Andreas Hallqvist +8Clinical Outcome Prediction Medical VLMs
Chalmers University of Technology, Gothenburg, Sweden
V-GIFT: Boosting Visual Instruction Tuning with Self-Supervised Guidance
Apr 14, 2026Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc +2Visual Instruction Tuning Large Vision-Language Models
Sorbonne Universite, CNRS, ISIR, F-75005 Paris, France · Institut universitaire de France (IUF)
A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models
Mar 31, 2026Lixin Xiu, Xufang Luo, Hideki NakayamaVisual Instruction Tuning Large Vision-Language Models
The University of Tokyo · Microsoft Research
Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning
Mar 12, 2026Rujie Wu, Haozhe Zhao, Hai Ci +1Multimodal Optimization Instruction-Tuning Data Selection
School of Computer Science, Peking University, Beijing, China · Institute for Artificial Intelligence, Peking University, Beijing, China · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China +2
VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning
Mar 1, 2026Mingkang Dong, Hongyi Cai, Jie Li +4Vision-Language Models Data Selection
SJTU · Universiti Malaya · CMU +3

Oct 14, 2026J/K move · Enter open · S save

Instruction Mining From Images: Constructing a Synthetic Dataset for Multimodal Learning ILSP DCVLR Competition Submission

Eleni E. Kasoura, Dimitrios Damianos, Georgios Paraskevopoulos, Vassilios Katsouros

Institute of Language and Speech Processing Athena Research Center Athens, GR 151 25

Visual Instruction Tuning Synthetic Data Generation for Language Models Synthetic Instruction Generation Multimodal Instruction Tuning

Open paper AskPDF arXiv

This work presents a compact and reproducible pipeline for synthesizing and curating image-based instruction-response pairs for DCVLR. We draw from the Cauldron collection and use Magpie-style instruction and response synthesis with Qwen2.5-VL-7B Instruct to create concise, image-conditioned instruction pairs. The pipeline consists of instruction synthesis, conditioned on input images and controlled via a system prompt, sanitization, tagging, and filtering of the instructions based on the score obtained by a reward model (DeBERTa-v3). We retain samples with positive scores to maximize the quality of the generated instructions. We observe that score distributions have a median and mean reward score of ~ 1.53 and ~ 1.75 respectively, indicating that most generated instructions are of good quality. The resulting dataset contains diverse instructions in the math, spatial, and social reasoning domains.