From Blind Spots to Gains: Diagnostic-Driven Iterative Training for Large Multimodal Models

Hongrui Jia; Chaoya Jiang; Yongrui Heng; Shikun Zhang; Wei Ye

cs.CVFeb 26, 2026

From Blind Spots to Gains: Diagnostic-Driven Iterative Training for Large Multimodal Models

Authors: Hongrui Jia, Chaoya Jiang, Yongrui Heng, Shikun Zhang, Wei Ye

Organizations: Peking University · Shandong University

Abstract

As Large Multimodal Models (LMMs) scale up and reinforcement learning (RL) methods mature, LMMs have made notable progress in complex reasoning and decision making. Yet training still relies on static data and fixed recipes, making it difficult to diagnose capability blind spots or provide dynamic, targeted reinforcement. Motivated by findings that test driven error exposure and feedback based correction outperform repetitive practice, we propose Diagnostic-driven Progressive Evolution (DPE), a spiral loop where diagnosis steers data generation and reinforcement, and each iteration re-diagnoses the updated model to drive the next round of targeted improvement. DPE has two key components. First, multiple agents annotate and quality control massive unlabeled multimodal data, using tools such as web search and image editing to produce diverse, realistic samples. Second, DPE attributes failures to specific weaknesses, dynamically adjusts the data mixture, and guides agents to generate weakness focused data for targeted reinforcement. Experiments on Qwen3-VL-8B-Instruct and Qwen2.5-VL-7B-Instruct show stable, continual gains across eleven benchmarks, indicating DPE as a scalable paradigm for continual LMM training under open task distributions. Our code, models, and data are publicly available at https://github.com/hongruijia/DPE.

Figures & tables

Figure 1: Due to the lack of interpretable diagnostics and scarcity of visual diversity, previous self-evolution frameworks can alleviate hallucination to some extent but fail to provide meaningful improvements on long-tail tasks such as mathematics and OCR. As a result, the model often exhibits instability or even degradation in these capabilities during the evolution process. In contrast, our DPE framework effectively addresses these blind spots and supports a more comprehensive and balanced progression of the model’s abilities.

Figure 2: Overview of the DPE framework.

Models		Qwen2.5-VL-7B-Instruct							Qwen3-VL-8B-Instruct
Methods		Base	VisPlay			DPE (Ours)			Base	DPE (Ours)
Methods		Base	Iter 1	Iter 2	Iter 3	Iter 1	Iter 2	Iter 3	Base	Iter 1	Iter 2	Iter 3
STEM	MMMU	53.11	53.33	49.3	54.89	54.44	55.33	56.44	65.44	68.11	69.11	69.11
	RealWorldQA	68.63	68.50	68.10	69.02	69.41	69.54	70.46	71.63	71.63	70.72	70.85
	MMVet	67.20	67.84	69.44	67.52	67.71	67.02	68.35	67.29	70.92	70.00	72.80
	MMStar	63.27	63.60	66.07	65.07	65.00	64.60	65.60	61.27	71.40	71.67	72.13

Table 1: DPE’s performance compared with Self-evolving Method’s.

Models	General Visual Understanding		Visual Math			OCR	Hallucination	Avg
Models	MMMU	MMStar	MathVerse	MathVision	MathVista	CharXiv RQ	HallusionBench	Avg
DeepEyes	-	-	47.3	26.6	70.1	-	-	-
DeepEyesV2	-	-	52.9	28.9	38.1	48.9	-	-
Qwen2.5-VL-72B	70.2	70.8	57.6	38.1	74.8	49.7	72.4	61.9
GPT-4o	69.1	64.7	50.2	30.4	63.8	47.1	67.5	56.1
GPT5-Mini	67.9	61.3	36.5	46.6	59.6	48.9	55.9	53.8

Table 2: DPE’s Performance Compared with State-of-the-Arts’.

Qwen2.5-VL-7B-Instruct
Methods	Data Size	MMMU	HallusionBench	MathVista	RealWorldQA
Vision-R1	47K	54.8	67.6	68.8	69.9
DPE (Iter 3)	3K	56.44	69.0	69.5	70.5

Table 3: Comparison between static training and DPE.

Figure 3: Ablation results on CharXiv and MathVision across three iterations, comparing full DPE with variants.

Figure 4: Category distribution of the seed set and the diagnosis-guided mixture ratios recommended by DPE over three iterations.

Figure 5: UMAP visualization of image diversity (left) and text diversity (right) for VisPlay and DPE.

Domain	Base	VisPlay			DPE (Ours)
Domain	Base	Iter 1	Iter 2	Iter 3	Iter 1	Iter 2	Iter 3
Text	0.764	0.830	0.820	0.797	0.846	0.866	0.850
Image	0.835	0.835	0.835	0.835	0.847	0.864	0.877

Table 4: Text and image diversity scores (mean pairwise cosine distance) using Qwen3-VL-Embedding.

Metric	VisPlay			DPE (Ours)
Metric	Iter 1	Iter 2	Iter 3	Iter 1	Iter 2	Iter 3
CL	4.38	4.14	3.91	4.99	4.86	4.92
S	3.45	3.58	2.98	4.98	4.86	4.91
CO	3.40	3.52	3.08	4.93	4.56	4.58
QS	3.74	3.75	3.32	4.96	4.74	4.80

Table 5: Quality evaluation of generated questions.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Case Study between VisPlay and our DPE framework.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

Learn to Think: Improving Multimodal Reasoning through Vision-Aware Self-Improvement Training
May 12, 2026Qihuang Zhong, Liang Ding, Wenjie Xuan +3Multimodal Large Language Models Multimodal Reasoning
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China · The University of Sydney, Australia · Nanyang Technological University, Singapore
How to Teach Large Multimodal Models New Skills
Oct 9, 2025Zhen Zhu, Yiming Gong, Yao Xiao +2Multimodal Large Language Models Continual Learning for VLMs
University of Illinois Urbana-Champaign · Google DeepMind · Carnegie Mellon University
MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training
Oct 4, 2026Yuxin Liu, Yuxuan Wang, Zhenxin Lei +9Multimodal Model Evaluation Agent Benchmarks
University of Science and Technology of China · Alibaba Token Hub, Alibaba Group · University of the Chinese Academy of Sciences +2

May 12, 2026J/K move · Enter open · S save

Learn to Think: Improving Multimodal Reasoning through Vision-Aware Self-Improvement Training

Qihuang Zhong, Liang Ding, Wenjie Xuan, Juhua Liu+2

School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China · The University of Sydney, Australia · Nanyang Technological University, Singapore

Multimodal Large Language Models Multimodal Reasoning VLM Reasoning Language Model Self-Improvement

Open paper AskPDF arXiv

Post-training with explicit reasoning traces is common to improve the reasoning capabilities of Multimodal Large Language Models (MLLMs). However, acquiring high-quality reasoning traces is often costly and time-consuming. Hence, the self-improvement paradigm has emerged, enabling MLLMs to self-generate reasoning traces for training without external supervision. Despite its effectiveness, we reveal two shortcomings in the self-improvement training of MLLMs: 1) data imbalance, where simple samples are over-trained, but the challenging yet crucial samples are under-trained; 2) language prior bias, where MLLMs overly rely on linguistic priors while neglecting the visual cues. To this end, we propose VISTA, a vision-aware self-improvement training framework for enhancing the multimodal reasoning of MLLMs. Specifically, VISTA first introduces a prefix resampling strategy to reuse the partial correct reasoning traces for efficient data collection, and then designs a vision-aware attention score to quantify the model's focus on visual information. Extensive experiments show that VISTA can be applied to various post-training scenarios, i.e., supervised fine-tuning and preference learning, and effectively enhances the multimodal reasoning performance across various MLLMs and tasks, e.g., bringing up to +13.66% average performance gains for Qwen2.5-VL-3B-Instruct.

Abstract

Multimodal Large Language Models

Explore similar work

Learn to Think: Improving Multimodal Reasoning through Vision-Aware Self-Improvement Training

How to Teach Large Multimodal Models New Skills

MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training