Multimodal instruction-following models require training data that is accurate, diverse, verifiable, and challenging. Existing synthesis pipelines typically follow a one-pass generate-and-filter paradigm, discarding feedback from failed samples, verifier outcomes, and target-model errors. We present VISA (Visual Instruction Synthesis Agent), an agentic framework that reformulates multimodal instruction synthesis as a self-evolving loop. At each round, VISA analyzes an image to filter incompatible constraints and discover new verifiable ones, samples diversity- and difficulty-aware constraint sets from persistent memory, generates candidate instructions, and verifies the resulting samples with executable tools and structured large language model judges. Failed samples trigger diagnostic-guided recovery, while accepted samples are probed against the target model to estimate difficulty. The resulting verifier signals and target-model failure profiles are written back to memory, allowing subsequent rounds to adaptively expand the constraint space, reduce template repetition, and focus on unresolved model weaknesses. The same verifier contracts further provide reward signals for reinforcement learning without a separately trained reward model. Experiments on MM-IFEval show that VISA consistently improves multimodal instruction following over strong baselines, while preserving general multimodal capability across seven public benchmarks.
Figures & tables
Figure 1: Overview of VISA. The framework iteratively performs perception, planning and execution, reflection, and memory update, allowing verifier feedback and target-model difficulty signals to guide subsequent synthesis rounds.
Model / Training
Params
MM-IFEval
C-Level
P-Level
Avg.
Qwen3.5-9B
9B
63.9
60.0
63.0
MiniCPM-V-4.5
8B
68.8
44.0
62.6
InternVL3.5-8B
8B
61.8
39.0
56.1
InternVL3.5-4B
4B
57.0
33.0
51.0
Qwen3.5-4B
4B
62.7
55.0
60.8
Table 1: Main results on MM-IFEval. We report C-Level accuracy, P-Level accuracy, and their average score; higher values indicate better instruction-following performance.
Method (Qwen3.5-4B)
C-Level
P-Level
Avg.
Base
62.7
55.0
60.8
Static Pipeline
64.9
46.0
60.2
+ Reflection
65.6
48.0
61.2
+ Reflection + Memory
67.1
51.0
63.1
Full VISA
67.2
54.0
63.9
Table 2: Ablation results on MM-IFEval with Qwen3.5-4B under SFT, showing the incremental contribution of reflection, memory, and the full VISA pipeline.
Model / Training
MMBench
MMStar
MM-Vet
HallusionBench
MathVista
OCRBench
AI2D
Avg.
Qwen3.5-4B
78.4
61.7
47.6
56.0
78.8
84.9
86.2
70.5
+ VISA-SFT-15k
81.2
63.2
49.1
54.1
77.8
85.2
86.4
71.0
+ VISA-RL-15k
78.8
62.5
50.5
57.5
87.9
85.6
87.6
72.9
Table 3: General multimodal capability results on Qwen3.5-4B. VISA preserves or improves broad multimodal performance across seven public benchmarks, with all evaluations conducted in non-thinking mode.
Figure 2: Evolution of the constraint-verifier registry in VISA. Left: cumulative growth of constraint types across self-evolution rounds, showing that most discoveries occur early and gradually saturate. Right: category distribution of the final 310 constraint types.
Figure 3: Diversity analysis of VISA-synthesized instructions. Left: conditional co-occurrence among the ten most frequent constraint categories, where each cell reports P(j∣i) as a percentage. Right: t-distributed stochastic neighbor embedding (t-SNE) projection of high-dimensional instruction embeddings, colored by dominant constraint category.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Distribution of the number of constraints per synthesized sample, showing that most accepted samples combine multiple jointly satisfiable requirements rather than a single constraint.
Table 4: Constraint categories in the final VISA registry. Representative types are illustrative examples; “Seed” and “+Disc.” denote initially defined and dynamically discovered constraint types, respectively.
Multimodal instruction following (MMIF) is crucial for building generalist agents. However, current training paradigms rely heavily on Supervised Fine-Tuning (SFT), which often leads to surface-level pattern matching and degrades general capabilities. While Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising alternative, its scalability in MMIF is severely bottlenecked by the scarcity of high-quality, RL-ready multimodal data. To bridge this gap, we present MIFS (Multimodal Instruction Following Synthesis), a systematic pipeline designed to generate RL-ready multimodal data. Specifically, MIFS introduces a generative constraint protocol to synthesize diverse raw samples, followed by a learnability-aware distillation mechanism that filters data based on RL training dynamics to ensure stable policy optimization. Furthermore, a code-based verifier provides high-precision reward signals for policy learning. The resulting dataset comprises 90k samples across 8 constraint categories and 14 task domains. Empirical evaluations demonstrate that MIFS-trained MLLMs achieve an average improvement of 8.13% on four MMIF benchmarks and a 3× faster training convergence compared to using raw data. Crucially, our approach mitigates the generalization trade-offs typical of SFT, preserving core visual capabilities while significantly boosting instruction-following precision.
Yirong Zeng, Zhang Sai, Yuxian Wang +4
Harbin Institute of Technology, SCIR Lab · Peking University
Multimodal Large Language Models (MLLMs) are increasingly being applied in a wider range of real-world scenarios. However, due to the substantial labor cost, creating high-quality multimodal instruction datasets for MLLMs remains a significant challenge. Although some methods propose to generate instruction data, they often face limitations in modality support and struggle with generating multi-round instructions. To address these problems, we introduce UniData, a universal instruction generation pipeline, to transform simple user requirements into multi-round, multimodal instructions. Specifically, UniData first expands user requirements into multiple diverse events. Using these events, UniData then integrates an any-to-any large model for multimodal instruction generation. Finally, UniData enhances data quality by correcting irrelevant and redundant inference flow, leveraging correlations between instruction rounds. To train this pipeline, we also build UniDataset, a dataset comprising 20,000 entries across nine modalities for improved multimodal generation. Our experiments demonstrate that UniData achieves SOTA performance in data quality and can also enhance the understanding and generation capabilities of other multimodal models.
Jiaqi Tang, Yi-Feng Wu, Yuting Zhang +12
The Hong Kong University of Science and Technology · ATH, Alibaba Group · The Hong Kong University of Science and Technology (Guangzhou) +2
Real-world multimodal instructions often bundle multiple requirements with unequal importance, yet most multimodal training data still reduce instruction following to answering one self-contained question. We study this gap through rubric comprehension, which casts the model not as a generator measured against rubrics but as an executor that follows them: given an image and a typed, prioritized rubric, the model must verify each rule before producing an overall judgment. To support this setting, we propose PRISM, a four-stage data synthesis framework that produces persona--task pairs, prefix-guided rule sets, quality-filtered rubrics, and structured verification traces. We further introduce PRISM-Eval, whose Loose and Strict metrics use deterministic matching against fixed labels and therefore require no inference-time judge model. With only 10K synthesized samples, PRISM lifts Qwen3-VL-4B from 9.5% to 30.1% Strict accuracy on PRISM-Eval while preserving average performance on general benchmarks, and the gains transfer to four additional open-source MLLMs across dense and MoE architectures, suggesting that structured rubric supervision is a scalable path toward multi-rule, priority-aware multimodal instruction following.