Instruction Mining From Images: Constructing a Synthetic Dataset for Multimodal Learning ILSP DCVLR Competition Submission

Eleni E. Kasoura; Dimitrios Damianos; Georgios Paraskevopoulos; Vassilios Katsouros

cs.CVOct 14, 2026

Instruction Mining From Images: Constructing a Synthetic Dataset for Multimodal Learning ILSP DCVLR Competition Submission

Authors: Eleni E. Kasoura, Dimitrios Damianos, Georgios Paraskevopoulos, Vassilios Katsouros

Organizations: Institute of Language and Speech Processing Athena Research Center Athens, GR 151 25

Abstract

This work presents a compact and reproducible pipeline for synthesizing and curating image-based instruction-response pairs for DCVLR. We draw from the Cauldron collection and use Magpie-style instruction and response synthesis with Qwen2.5-VL-7B Instruct to create concise, image-conditioned instruction pairs. The pipeline consists of instruction synthesis, conditioned on input images and controlled via a system prompt, sanitization, tagging, and filtering of the instructions based on the score obtained by a reward model (DeBERTa-v3). We retain samples with positive scores to maximize the quality of the generated instructions. We observe that score distributions have a median and mean reward score of ~ 1.53 and ~ 1.75 respectively, indicating that most generated instructions are of good quality. The resulting dataset contains diverse instructions in the math, spatial, and social reasoning domains.

Figures & tables

Figure 1: The proposed pipeline for data synthesis and curation based on Magpie.

Figure 2: Reward score distribution of selected samples

Subset	Count
cocoqa	1354
aokvqa	1311
vqav2	1289
clevr	998
ocrvqa	971
ai2d	695

Table 1: Subset counts for sampes included in the generated dataset. The dataset includes 10000 samples in total.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Prompt type	Prompt examples (2 per type)
prompts	Try to ask an easy question for this image.
	Create an instruction that prompts the user to explain the sequence of events taking place in the image.
math_prompts	Create a question that asks the user to describe the mathematical concept represented in the diagram.
	Write an instruction that asks the user to summarize the main statistical findings illustrated by this chart.
multi_image	Compare the 2 images and write a question that highlights the differences in the objects or scenes depicted.
	Look at the 2 images and create an instruction that asks the user to identify the similarities between the two.

Appendix

Table 2: Prompt categories with two example prompts per type.

Visual Instruction Tuning

See all 12 papers

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
Jan 29, 2026Ajay Patel, Colin Raffel, Chris Callison-BurchLanguage Model Pretraining Synthetic Data Pretraining
University of Pennsylvania · University of Toronto · Vector Institute +1
VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following
Aug 26, 2026Min Zeng, Guanxin Tan, Libin Cen +5Multimodal Instruction Tuning
vivo AI Lab
EditCaption: Human-Refined SFT and HAE-DPO for Image Editing Instruction Synthesis
Apr 9, 2026Xiangyuan Wang, Honghao Cai, Yunhao Bai +9Direct Preference Optimization Image Captioning
Peking University · Xiaohongshu Inc. · The Chinese University of Hong Kong, Shenzhen +2

Jan 29, 2026J/K move · Enter open · S save

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Ajay Patel, Colin Raffel, Chris Callison-Burch

University of Pennsylvania · University of Toronto · Vector Institute+1

Language Model Pretraining Synthetic Data Pretraining Synthetic Data Generation Synthetic Instruction Generation Instruction Tuning Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .

Abstract

Visual Instruction Tuning

Explore similar work

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following

EditCaption: Human-Refined SFT and HAE-DPO for Image Editing Instruction Synthesis