Data Organization Matters in Multimodal Instruction Tuning: A Controlled Study of Capability Trade-offs

Guowei Tang

cs.CVMar 29, 2026

Data Organization Matters in Multimodal Instruction Tuning: A Controlled Study of Capability Trade-offs

Authors: Guowei Tang

Organizations: School of Statistics, East China Normal University

Abstract

Recent multimodal large language models (MLLMs) perform strongly on general visual understanding, diagram and chart reasoning, and document-centric perception. However, these abilities are learned from heterogeneous supervision sources with very different task structures and learning demands, and the effect of their temporal organization during training remains underexplored. We study whether data organization affects the trade-off among general understanding, structured reasoning, and fine-grained OCR/document understanding in multimodal instruction tuning. To isolate this factor, we use a controlled three-stage training framework in which the backbone, trainable modules, and optimization pipeline are fixed across all runs, and only the temporal arrangement of post-alignment supervision is changed. We compare four strategies: direct mixture, curriculum training, balanced sampling, and reverse curriculum. Experiments on general visual instruction following, diagram reasoning, chart reasoning, scene-text question answering, and document question answering show that data organization is a first-order design variable in multimodal adaptation. Curriculum training gives the best overall trade-off and the strongest structured reasoning performance. Balanced sampling is better for OCR-oriented capability but weakens the broader capability balance. Reverse curriculum performs worst in both final performance and optimization stability. Training-dynamics analysis further suggests that building general understanding and reasoning before introducing OCR-intensive supervision leads to smoother optimization and faster convergence. These findings highlight data scheduling as an explicit design dimension for multimodal model adaptation.

Figures & tables

Condition	Strategy	Stage 1	Stage 2	Stage 3
A	Direct mixture	$D_{0}$ only	S4V: 0.50, AI2D: 0.13, ChartQA: 0.13, TXT: 0.12, DOC: 0.12	S4V: 0.50, AI2D: 0.13, ChartQA: 0.13, TXT: 0.12, DOC: 0.12
B	Curriculum	$D_{0}$ only	S4V: 0.70, AI2D: 0.15, ChartQA: 0.15	S4V: 0.20, AI2D: 0.10, ChartQA: 0.10, TXT: 0.30, DOC: 0.30
C	Balanced sampling	$D_{0}$ only	S4V: 0.20, AI2D: 0.20, ChartQA: 0.20, TXT: 0.20, DOC: 0.20	S4V: 0.20, AI2D: 0.20, ChartQA: 0.20, TXT: 0.20, DOC: 0.20
D	Reverse curriculum	$D_{0}$ only	S4V: 0.20, AI2D: 0.10, ChartQA: 0.10, TXT: 0.30, DOC: 0.30	S4V: 0.70, AI2D: 0.15, ChartQA: 0.15

Table 1: Summary of stage-wise data organization strategies. S4V denotes ShareGPT4V, TXT denotes TextVQA, and DOC denotes DocVQA.

Condition	General	Reasoning			OCR			Overall
Condition	General-Val	AI2D	ChartQA	Reasoning	TextVQA	DocVQA	OCR	Overall
A	72.1	74.0	72.0	73.0	70.8	71.4	71.1	72.1
B	73.4	76.0	74.6	75.3	71.8	72.9	72.4	73.7
C	71.7	73.3	72.9	73.1	72.6	73.5	73.1	72.8
D	69.8	72.1	70.9	71.5	71.4	71.8	71.6	71.2

Table 2: Main results on the validation benchmarks under the proposed data-organization scheme. Higher is better. The best score in each column is bolded .

Figure 1: Comparison of all the scores across different training conditions.

Figure 2: Comparison of Overall across different training conditions.

Condition	Loss Std $\downarrow$	Spike Freq. $\downarrow$	Stage Tran Stab
A	0.117	2.12%	0.29
B	0.096	1.48%	0.37
C	0.109	1.84%	0.27
D	0.136	2.73%	0.68

Table 3: Training dynamics and efficiency statistics. Lower is better for loss standard deviation, spike frequency, stage-transition loss gap.

Explore similar work

MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning
Oct 1, 2026Juekai Lin, Honglin Lin, Yuqian Yuan +7Reinforcement Learning Multimodal Reasoning
Zhejiang University · Shanghai Artificial Intelligence Laboratory, OpenDataLab · Shanghai Jiao Tong University +1
VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning
Mar 1, 2026Mingkang Dong, Hongyi Cai, Jie Li +4Vision-Language Models Data Selection
SJTU · Universiti Malaya · CMU +3
Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning
Oct 11, 2026Quanlin Chen, Xiuwei Chen, Kun Xiang +5Multimodal Large Language Models Multimodal Reasoning
Sun Yat-sen University · Shenzhen Loop Area Institute · Beijing Zhongguancun Academy +2

Oct 1, 2026J/K move · Enter open · S save

MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning

Juekai Lin, Honglin Lin, Yuqian Yuan, Xiaolong Wu+6

Zhejiang University · Shanghai Artificial Intelligence Laboratory, OpenDataLab · Shanghai Jiao Tong University+1

Reinforcement Learning Multimodal Reasoning Multimodal Instruction Tuning

Open paper AskPDF arXiv

Open multimodal reasoning models have benefited from large-scale reasoning supervision, yet reliable post-training remains challenging due to uneven data quality, inefficient supervision construction, imbalanced difficulty, and cross-domain interference. We introduce MMVistaReason (MVR), an open-data post-training recipe with three components: (1) broader capability coverage across complementary Analytical and Real-World reasoning groups, emphasizing structured reasoning versus visual perception and spatial grounding; (2) efficient SFT and RL data construction, standardizing heterogeneous open data through staged cleaning and annotation, combining difficulty-aware cascaded teacher distillation with answer-likelihood-based trajectory selection to construct MVR-SFT-528K, and applying scale-specific frontier filtering for MVR-RL-63K; and (3) specialize-then-integrate training, which trains complementary RL experts and consolidates their capabilities through multi-teacher on-policy distillation (MOPD). Our analyses reveal a capacity-dependent interaction between supervision difficulty, trajectory quality, and model capacity: smaller students benefit more from selected supervision, while larger students are robust to trajectory variation and mixed-domain interference. Mixed-domain RL introduces benchmark-level negative transfer, whereas MOPD provides consistent capability integration, with the preferred KL direction varying across model scales. Across 15 multimodal benchmarks, MVR-4B achieves an average score of 72.8, outperforming Qwen3.5-9B (Instruct) and MMFineReason-8B while using about 70% fewer samples than MMFineReason. Scaling to 9B improves the average to 74.4, surpassing Qwen3.5-35B-A3B (Instruct). Overall, MMVistaReason demonstrates that systematic open-data construction and capacity-aware post-training provide a practical and scalable path toward reliable multimodal reasoning.

Abstract

Explore similar work

MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning

VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning

Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning