Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillation

Yirong Zeng; Zhang Sai; Yuxian Wang; Yutai Hou; Yufei Liu; Xiao Ding; Bibo Cai

cs.CLSep 13, 2026

Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillation

Authors: Yirong Zeng, Zhang Sai, Yuxian Wang, Yutai Hou, Yufei Liu, Xiao Ding, Bibo Cai

Organizations: Harbin Institute of Technology, SCIR Lab · Peking University

Abstract

Multimodal instruction following (MMIF) is crucial for building generalist agents. However, current training paradigms rely heavily on Supervised Fine-Tuning (SFT), which often leads to surface-level pattern matching and degrades general capabilities. While Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising alternative, its scalability in MMIF is severely bottlenecked by the scarcity of high-quality, RL-ready multimodal data. To bridge this gap, we present MIFS (Multimodal Instruction Following Synthesis), a systematic pipeline designed to generate RL-ready multimodal data. Specifically, MIFS introduces a generative constraint protocol to synthesize diverse raw samples, followed by a learnability-aware distillation mechanism that filters data based on RL training dynamics to ensure stable policy optimization. Furthermore, a code-based verifier provides high-precision reward signals for policy learning. The resulting dataset comprises 90k samples across 8 constraint categories and 14 task domains. Empirical evaluations demonstrate that MIFS-trained MLLMs achieve an average improvement of 8.13% on four MMIF benchmarks and a 3 $\times$ faster training convergence compared to using raw data. Crucially, our approach mitigates the generalization trade-offs typical of SFT, preserving core visual capabilities while significantly boosting instruction-following precision.

Figures & tables

Dataset	#Sample	#Constraint	#Domain	Eval?	Evaluator	SFT?	RLVR?
MIA-BENCH ( Qian et al. )	400	8	15	✓	LLM	✗	✗
CrafText ( Volovikova et al. 2025 )	3,924	-	4	✓	Rule	✗	✗
MMMT-IF ( Sun et al. 2025 )	990	4	13	✓	Rule	✗	✗
MM-IFInstruct ( Ding et al. 2025 )	23k	32	13	✓	LLM&Rule	✓	✗
MIFS (Ours)	90,838	8	14	✓	Rule	✓	✓

Table 1: Comparison of Multimodal Instruction Following (MM-IF) datasets. MIFS distinguishes itself by providing the largest scale of programmatically verifiable samples, specifically engineered for RLVR beyond simple SFT.

Figure 1: The overall framework of the Multimodal Instruction Synthesis suitable for both SFT and RL training. It comprises: (1) Instruction Composition, where the raw multimodal IF and their verifier are synthesized via LLMs; (2) Multi-Step Quality Distillation, featuring a three-tier filtering mechanism that prunes samples based on constraint density, model-perceived difficulty, and RL training dynamics; and (3) Sample Curation, illustrating data characteristics and partitioning strategy.

Figure 2: RL training dynamic filtering. Retaining aligned samples (i.e., those with low variance (a) ) and removing outlier samples (i.e., those with high variance (b)).

Figure 3: Empirical analysis of multi-step quality distillation. It showcases: (a) Stepwise contraction of data scale across steps; (b) Monotonic alignment gains in RL training;

Figure 4: The distribution of constraint types. The dataset comprises 8 verifiable constraint categories with an average density of 7.1 constraints per sample, ensuring high-constraint complexity.

Split	# Samples	Jaccard $\geq 0.8$	Cosine $\geq 0.8$
MIFS-SFT	83,280	0.00%	2.61%
MIFS-RL	60,048	0.00%	4.73%
MIFS-Eval	1,200	0.00%	1.31%

Table 2: Semantic overlap and near-duplicate rates across dataset splits. Minimizing instance redundancy prevents shortcut, safeguarding policies during RLVR training.

Figure 5: The hierarchical distribution of task diversity. The chart features a nested structure: the inner ring classifies tasks into four high-level cognitive domains, segment size reflects task frequency.

Qwen3-VL-4B
Method	Benchmarks				Average
Method	MIABench	MM-IFEval	MIFS-Eval	IFEval	Average
Base	$92.2_{\pm 0.2}$	$58.5_{\pm 0.4}$	$72.8_{\pm 0.4}$	$79.1_{\pm 0.2}$	75.6
MIFS-RL	$95.5_{\pm 0.1}$	$66.8_{\pm 0.5}$	$85.5_{\pm 0.4}$	$87.4_{\pm 0.3}$	83.8 ↑8.2
Qwen3-VL-8B
Base	$91.2_{\pm 0.3}$	$62.7_{\pm 0.5}$	$74.5_{\pm 0.3}$	$81.2_{\pm 0.2}$	77.4

Table 3: Performance comparison on Instruction Following benchmarks. We report scores for MIABench, MM-IFEval, MIFS-Eval, and IFEval, along with their group-wise average. All metrics are reported as the

mean_{\pm std}

over 3 independent runs.

Qwen3-VL-4B
Method	OCR	MMVet	MMBen	Avg.
Base Model	86.50	65.60	84.41	78.84
MIFS-RL	87.52	67.32	83.24	79.36 ↑0.5
Qwen3-VL-8B
Base Model	88.00	69.72	85.30	81.01
MIFS-SFT	78.62	57.32	70.30	68.75 ↓12.3

Table 4: Performance on General Visual Capability benchmarks (i.e., OCRBench, MMVet, and MMBench).

Panel of Figure 6 — Figure 6: Training process visualization: (a) Raw data and (b) final MIFS-RL data. The MIFS-RL curriculum demonstrates significantly faster convergence ( $\sim 3\times$ ) and higher test performance.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Value
Optimizer	AdamW ( $\beta_{1}=0.9,\beta_{2}=0.95$ )
Learning Rate	$2\times 10^{-5}$
Learning Rate Scheduler	Cosine decay
Weight Decay	0.1
Global Batch Size	128
Precision	BFloat16

Appendix

Table 5: Hyper-parameters for SFT training.

Data & Rollout
Hyperparameter	Value
Global Batch Size	128
Max Prompt/Resp.	2k / 4k
Sampling Temp. $\tau$	1.0
Top- $p$	0.6
Rollout Group Size $G$	8

Appendix

Table 6: Hyperparameters for GRPO training.

	Code: Pass	Code: Fail
Human: Pass	35 (TP)	3 (FN)
Human: Fail	1 (FP)	11 (TN)
Metrics	Agreement: 92.0% \| Precision: 97.22%

Appendix

Table 7: Confusion Matrix: Automated Verifier vs. Human Judgment (N=50).

Constraint Type	Definition	Example
Keyword Inclusion	Mandatory integration of specific lexical items.	“Must include the words ’sustainability’ and ’innovation’.”
Negative Constraint	Prohibiting the use of specific words, phrases, or topics.	“Do not use the word ’very’ or mention price details.”
Length Constraint	Quantifiable limits on the number of characters, words, or sentences.	“The response must be between 50 and 80 words long.”
Structural Constraint	Requirements regarding the organizational layout or components.	“Format the response into three distinct bullet points.”
Punctuation Constraint	Rules governing the usage or exclusion of specific punctuation marks.	“Each sentence must end with a semicolon instead of a period.”
Positioning Constraint	Placing specific content at predefined locations within the text.	“The output must conclude with the phrase ’End of Report’.”

Appendix

Table 8: Definitions and Examples of Constraint Types in our Dataset.

Qwen3-8B (Text-Only)
Dataset / Method	MIABench	MM-IFEval	MIFS-Eval	IFEval	Average
Base	-	-	-	85.77	-
RECAST-SFT	-	-	-	87.14	-
RECAST-RL	-	-	-	88.65	-
VERIF-RL	-	-	-	86.51	-
Qwen3-VL-8B (Multimodal)

Appendix

Table 9: Comparison with additional baselines. MIFS demonstrates superior performance in the multimodal setting compared to both standard SFT and text-only RL approaches.

Explore similar work

VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following
Aug 26, 2026Min Zeng, Guanxin Tan, Libin Cen +5Multimodal Instruction Tuning
vivo AI Lab
UniData: Universal Multimodal Instruction Generation Pipeline
Oct 8, 2026Jiaqi Tang, Yi-Feng Wu, Yuting Zhang +12Synthetic Data Generation for Language Models Multimodal Generation
The Hong Kong University of Science and Technology · ATH, Alibaba Group · The Hong Kong University of Science and Technology (Guangzhou) +2
Mitigating Exploration Bias in RL for Multi-Instruction Following
Aug 24, 2026Mian Zhang, Yueqin Yin, Kaiyu He +4Instruction Following RL for Language Models
UT Dallas · UT Austin · UC Santa Barbara

Aug 26, 2026J/K move · Enter open · S save

VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following

Min Zeng, Guanxin Tan, Libin Cen, Yafei Wen+4

vivo AI Lab

Multimodal Instruction Tuning

Open paper AskPDF arXiv

Multimodal instruction-following models require training data that is accurate, diverse, verifiable, and challenging. Existing synthesis pipelines typically follow a one-pass generate-and-filter paradigm, discarding feedback from failed samples, verifier outcomes, and target-model errors. We present VISA (Visual Instruction Synthesis Agent), an agentic framework that reformulates multimodal instruction synthesis as a self-evolving loop. At each round, VISA analyzes an image to filter incompatible constraints and discover new verifiable ones, samples diversity- and difficulty-aware constraint sets from persistent memory, generates candidate instructions, and verifies the resulting samples with executable tools and structured large language model judges. Failed samples trigger diagnostic-guided recovery, while accepted samples are probed against the target model to estimate difficulty. The resulting verifier signals and target-model failure profiles are written back to memory, allowing subsequent rounds to adaptively expand the constraint space, reduce template repetition, and focus on unresolved model weaknesses. The same verifier contracts further provide reward signals for reinforcement learning without a separately trained reward model. Experiments on MM-IFEval show that VISA consistently improves multimodal instruction following over strong baselines, while preserving general multimodal capability across seven public benchmarks.

Abstract

Explore similar work

VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following

UniData: Universal Multimodal Instruction Generation Pipeline

Mitigating Exploration Bias in RL for Multi-Instruction Following