When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning

Zhengxian Wu; Kai Shi; Chuanrui Zhang; Zirui Liao; Jun Yang; Ni Yang; Qiuying Peng; Luyuan Zhang

cs.CVMar 22, 2026

When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning

Authors: Zhengxian Wu, Kai Shi, Chuanrui Zhang, Zirui Liao, Jun Yang, Ni Yang, Qiuying Peng, Luyuan Zhang,

Organizations: OPPO AI Center · Tsinghua University · Nanyang Technological University · Hefei University of Technology

Abstract

Recent progress in multimodal large language models has led to strong performance on reasoning tasks, but these improvements largely rely on high-quality annotated data or teacher-model distillation, both of which are costly and difficult to scale. To address this, we propose an unsupervised self-evolution training framework for multimodal reasoning that achieves stable performance improvements without using human-annotated answers or external reward models. For each input, we sample multiple reasoning trajectories and jointly model their within group structure. We use the Actor's self-consistency signal as a training prior, and introduce a bounded Judge based modulation to continuously reweight trajectories of different quality. We further model the modulated scores as a group level distribution and convert absolute scores into relative advantages within each group, enabling more robust policy updates. Trained with Group Relative Policy Optimization (GRPO) on unlabeled data, our method consistently improves reasoning performance and generalization on five mathematical reasoning benchmarks, offering a scalable path toward self-evolving multimodal models. The code are available at https://github.com/OPPO-Mente-Lab/LLM-Self-Judge.

Figures & tables

Figure 1: Limitation of majority voting in unsupervised self-evolution.Right: An example where the most frequent answer is incorrect. Majority voting reinforces this dominant error, while our method favors higher-quality reasoning paths through Judge modulation.Left: Results on MathVision Wang et al. (2024) and DynaMath Zou et al. (2024) show that our approach consistently outperforms majority-voting-based self-training.

Figure 2: Overview of the proposed unsupervised self-evolution framework. The Actor generates multiple reasoning trajectories for the same input, while a frozen Judge provides bounded score modulation. The final rewards are optimized in a group-wise, distributional manner to enable stable policy updates without external supervision.

Performance on Qwen2.5-VL-7B
Training Data	Method	MathVision	MathVerse	WeMath	LogicVista	DynaMath	Avg.
—	Qwen2.5-VL-7B	25.0	44.2	37.1	46.3	20.3	34.6
Supervised Training & Teacher Model Distillation
Mixed Data	R1-Onevision-7B Yang et al. (2025b)	29.9	46.4	35.8	46.5	21.8	36.1
	OpenVLThinker-8B Deng et al. (2025)	25.9	50.3	36.5	46.3	21.2	36.0
	Vision-R1 Huang et al. (2025b)	29.4	52.4	35.4	49.7	25.2	38.4

Table 1: Main results on multimodal mathematical reasoning benchmarks. We report accuracy (%) on five math benchmarks. MajorVote corresponds to the MM-UPT method. denotes supervised training.

Method	MathVision	DynaMath	Avg.
Qwen2.5-VL-7B	25.0	20.3	22.65
+ Major Vote	27.5	21.4	24.45 +1.8
+ Self-Consistency	25.2	20.5	22.85 +0.2
+ Judge Scoring	27.3	21.1	24.20 +1.6
+ MV + JS	28.4	22.7	25.55 +2.9
+ SC + JS	30.1	23.7	26.90 +4.3

Table 2: Ablation experiments on different modules.

Model / Method	MathVision	DynaMath	Model / Method	MathVision	DynaMath
Qwen2-VL-2B	12.8	6.8	Qwen3-VL-8B	53.0	48.2
w/ major vote	9.6	4.2	w/ major vote	51.7	46.8
w/ ours	16.3	11.4	w/ ours	53.5	50.3
Qwen2.5-VL-3B	19.5	18.2	GLM-4.1V-9B	47.8	38.6
w/ major vote	21.3	19.1	w/ major vote	48.2	37.2
w/ ours	24.8	21.3	w/ ours	50.4	40.8

Table 3: Self-evolution performance comparison across different backbone models.

Figure 3: Training dynamics on MMR1 Leng et al. (2025) . The figure compares majority voting, supervised reinforcement learning, and our method during training, in terms of validation accuracy on MathVision, actor entropy, and average response length.

Figure 4: Ablation training dynamics on MMR1 Leng et al. (2025) . We compare Self-Consistency, Judge-only, and the full method in terms of validation accuracy on MathVision, actor entropy, and average response length during training.

Method	MathVision	DynaMath	Avg.
Qwen2.5-VL-7B	0.66	0.48	0.57
GRPO (w. label)	0.63	0.43	0.53 -0.04
Major Vote	0.58	0.37	0.48 -0.09
Ours	0.64	0.43	0.54 -0.03

Table 4: Comparison of pass@10 across benchmarks.

Model / Method	MathVision	LogicVista	DynaMath
Vision-R1-7B	29.4	49.7	25.2
w/ major vote	30.0	49.4	24.6
w/ ours	32.6	51.5	26.3

Table 5: Self-evolution results on a strong baseline that has already been trained with teacher distillation.

Method	Relative Time	Accuracy
Supervised GRPO	1.0 $\times$	29.1
MM-UPT (Major Vote)	1.0 $\times$	27.5
VisionZero	2.8 $\times$	27.6
EvoLMM	2.2 $\times$	25.8
Ours	1.4 $\times$	30.9

Table 6: Comparison of relative training cost and MathVision accuracy across different methods.

Training Setting	Method	ChartQA	MMVP
Base model	—	85.7	77.2
Geo3K	w/ MM-UPT	84.8	76.4
Geo3K	w/ ours	85.4	77.6
GeoQA	w/ MM-UPT	82.6	77.4
GeoQA	w/ ours	86.3	77.9
MMR1	w/ MM-UPT	85.4	74.6

Table 7: Generalization to chart understanding (ChartQA) and general visual reasoning (MMVP).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Hyperparameter (Value)
Data Configuration
Train Batch Size	512
Max Prompt Length	1024
Max Response Length	2048
Filter Overlong Prompts	True
Truncation Strategy	error

Appendix

Table 8: Training hyperparameters.

Figure 5: A case study on Geo3K Lu et al. (2021)

Figure 6: A case study on Geo3K Lu et al. (2021)

Panel of Figure 7 — Figure 7: A qualitative failure case analysis on the Geo3K dataset Lu et al. (2021) .

Method	MathVision	DynaMath
Self-Consistency	25.2	20.5
w. distribution	25.9	20.7
Judge	27.3	21.1
w. distribution	27.8	21.5
Ours w.o. distribution	30.1	23.7
Ours w. distribution	30.9	24.2

Appendix

Table 9: Effect of group-wise distributional modeling.

Method	Agree@1	SC-winner	J-winner
Qwen2.5-VL-7B	41.2	36.3	40.2
Ours	73.8	48.6	50.3

Appendix

Table 10: Relationship between self-consistency and Judge preferences.

Method	Overall	Algebra	Analytic Geo.	Arithmetic	Comb. Geo.
Qwen2.5-VL-7B	25.0	15.8	26.3	36.8	15.8
Ours	30.9	15.8	42.1	36.8	21.1
Method	Combinatorics	Counting	Descriptive Geo.	Graph Theory	Logic
Qwen2.5-VL-7B	10.5	21.1	26.3	15.8	15.8
Ours	21.1	21.1	21.1	10.5	21.1
Method	Metric (Angle)	Metric (Area)	Metric (Length)	Solid Geo.	Statistics

Appendix

Table 11: Category-wise overall accuracy (%) on MathVision (trained on Geo3K).

Method	Text Dominant	Vision Only	Vision Dominant	Vision Intensive	Text Lite
Qwen2.5-VL-7B	53.6	41.0	40.9	40.9	44.9
Ours	56.6	42.5	42.6	44.8	47.7

Appendix

Table 12: Category-wise overall accuracy (%) on MathVerse (trained on Geo3K).

Method	Overall	Inductive	Deductive	Numerical	Spatial	Mechanical
Qwen2.5-VL-7B	46.3	36.4	64.5	55.8	19.2	54.1
Ours	49.0	29.0	65.6	65.3	30.8	55.4

Appendix

Table 13: Category-wise overall accuracy (%) on LogicVista(trained on Geo3K).

Category	Qwen2.5-VL-7B	Ours	$\Delta$
Two-step (S2)	55.28	58.61	+3.33
Understanding of Plane Figures	60.22	64.45	+4.23
Calculation of Solid Figures	77.26	79.03	+1.77
Direction	82.86	90.00	+7.14
Route Map	55.49	62.64	+7.15

Appendix

Table 14: Category-wise overall accuracy (%) on Wemath (trained on Geo3K).

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 508 papers

AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution
May 25, 2026Zehao Wang, Yihan Zeng, Zidong Gong +5RL for Language Model Reasoning Multimodal Large Language Models
Harbin Institute of Technology · Huawei Noah’s Ark Lab · Independent Researcher
Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
Aug 18, 2026Yunhao Yang, Yuexin Bian, Yunjie Tian +6RL for Language Model Reasoning Reinforcement Learning
Johns Hopkins University · UC San Diego · Independent Researcher +1
A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
Oct 21, 2025Mengqi Li, Lei Zhao, Anthony Man-Cho So +2Self-Training for Language Models Mathematical Reasoning
The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University · The Chinese University of Hong Kong

May 25, 2026J/K move · Enter open · S save

AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution

Zehao Wang, Yihan Zeng, Zidong Gong, Yuanfan Guo+4

Harbin Institute of Technology · Huawei Noah’s Ark Lab · Independent Researcher

RL for Language Model Reasoning Multimodal Large Language Models Multimodal Reasoning

Open paper AskPDF arXiv

Post-training via Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) is crucial for enhancing reasoning in Multimodal Large Language Models (MLLMs), yet existing paradigms often reach a performance bottleneck due to the limitations of static data. While current methods leverage self-reflection or self-evolution to push these boundaries, they still suffer from cognitive drift and hallucinated reasoning paths caused by low-quality synthetic data. To address these challenges, we propose Anchor Evolution (AnE), a new paradigm that integrates truth-anchored data curation and model evolution, achieving faithful and steady performance gains at the reasoning frontier. Specifically, we propose Truth Anchor Expansion, which pinpoints the model failing frontier via trajectory rollouts and leverages ground-truth databases to retrieve high-fidelity anchors for faithful data curation. Subsequently, we introduce the Scaffold-Stripping Mechanism to internalize reasoning capabilities. This mechanism first anchors reasoning paths via scaffold-augmented supervision to mitigate the learning complexity and distribution drift of direct SFT on raw data, then leverages RL to strip the scaffold template, thereby effectively transitioning the reasoning paths into intrinsic model capabilities. Experimental results on multimodal reasoning benchmarks show that our method substantially advances the model performance frontier, improving the base model by 10.3% across eight multimodal benchmarks and achieving state-of-the-art results. The code will be made publicly available.

Abstract

Multimodal Reasoning

Explore similar work

AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning