Stabilizing Unsupervised Self-Evolution of MLLMs via Continuous Softened Retracing reSampling

Yunyao Yu; Zhengxian Wu; Zhuohong Chen; Hangrui Xu; Zirui Liao; Xiangwen Deng; Zhifang Liu; Senyuan Shi

cs.CVApr 4, 2026

Stabilizing Unsupervised Self-Evolution of MLLMs via Continuous Softened Retracing reSampling

Authors: Yunyao Yu, Zhengxian Wu, Zhuohong Chen, Hangrui Xu, Zirui Liao, Xiangwen Deng, Zhifang Liu, Senyuan Shi,

Organizations: Tsinghua University · Hefei University of Technology · University of Arizona · MAIS, Institute of Automation, Chinese Academy of Sciences

Abstract

In the unsupervised self-evolution of Multimodal Large Language Models, the quality of feedback signals during post-training is pivotal for stable and effective learning. However, existing self-evolution methods predominantly rely on majority voting to select the most frequent output as the pseudo-golden answer, which may stem from the model's intrinsic biases rather than guaranteeing the objective correctness of the reasoning paths. To counteract the degradation, we propose Continuous Softened Retracing reSampling (CSRS) in MLLM self-evolution. Specifically, we introduce a Retracing Re-inference Mechanism (RRM) that the model re-inferences from anchor points to expand the exploration of long-tail reasoning paths. Simultaneously, we propose Softened Frequency Reward (SFR), which replaces binary rewards with continuous signals, calibrating reward based on the answers' frequency across sampled reasoning sets. Furthermore, incorporated with Visual Semantic Perturbation (VSP), CSRS ensures the model prioritizes mathematical logic over visual superficiality. Experimental results demonstrate that CSRS significantly enhances the reasoning performance of Qwen2.5-VL-7B on benchmarks such as MathVision. We achieve state-of-the-art (SOTA) results in unsupervised self-evolution on geometric tasks. Our code is avaible at https://github.com/yyy195/CSRS.

Figures & tables

Figure 1: Overview of our work. During unsupervised self-evolutionary reinforcement learning, traditional majority voting methods (left) for pseudo-labeling rely solely on the model’s inherent biases. This frequently leads to model collapse, where the model degenerates into a deterministic mapping and fails to explore the true solutions. To alleviate this phenomenon, our method introduces CSRS (right) to reduce the occurence of situations.

Figure 2: Pipeline of our method . (Round1) illustrate initially maternal trajectories and answers generated by MLLM. (Round2) shows three core components of CSRS and improvement to rewards.

Model and Methods	Unsupervised?	Training Data	MathVision	MathVerse	MathVista	WeMath	Avg
Qwen2.5-VL-3B	✗	-	22.47	34.54	62.30	57.53	44.21
Qwen2.5-VL-7B	✗	-	25.40	44.24	66.42	67.65	50.93
MM-UPT	✓	Geometry3K	26.95	44.53	66.47	68.49	51.61
MM-UPT	✓	GeoQA	26.61	44.15	65.84	68.25	51.21
MM-UPT	✓	MMR1	25.98	45.12	66.27	69.14	51.63
CSRS (Ours)	✓	Geometry3K	27.97	46.01	67.81	71.77	53.39

Table 1: Comparison with supervised methods. Our unsupervised method CSRS shows competitive performance against base models and current SOTA MM-UPT across various benchmarks.

Table 4

Panel of Figure 3 — Figure 3: Visualization of the accuracy and propotion of high-confidence samples during training . (a) Answer accuracy calculated within partitioned frequency intervals, where each sample is uniquely assigned based on its frequency. (b) The evolution of high-confidence (frequence $\in$ $[0.8-1.0]$ ) samples proportions during training, which serves as a key indicator of model collapse.

Figure 4: The change of distribution of Maternal Answers Set and Re-inference Answers Set . The figure shows different distribution of samples in two sets at the step of 20,60,100. Blue dots represent the semantic space of vanilla responses, while orange dots denote the semantic space of responses generated by CSRS.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Entropy Change during training

Module	M-Vis.	M-Ver.	M-Vist.	WeMath
MajorityVote	25.40	44.24	66.42	67.65
+SFR	26.52	45.10	67.15	69.42
+RRM	26.85	45.35	67.38	69.85
+VSP	25.55	44.35	66.55	67.82
+SFR+RRM	27.68	45.85	67.62	71.30
+SFR+VSP	26.70	45.22	67.28	69.65

Appendix

Table 5: Ablation studies of different modules. The results demonstrate the incremental contribution of each component to the overall performance.

Data Configuration
Category	Hyperparameter (Value)
Train Batch Size	256
Max Prompt Length	1024
Max Response Length	1536
Filter Overlong Prompts	True
Truncation Strategy	"error"

Appendix

Table 6: Complete Hyperparameter and System Configurations.

Part 1: Performance of Base Models and Trained Models in Geometry3k
Model / Datasets	MathVision	MathVerse	MathVista	WeMath	Avg.
Qwen2.5-VL-7B(Vanilla)	25.40	44.24	66.42	67.65	50.93
+CSRS	27.97	46.01	67.81	71.77	53.39
InternVL3-8B(Vanilla)	29.30	45.12	71.60	73.25	54.82
+CSRS	30.58	47.23	72.98	75.15	56.49
ThinkLite-VL-7B(Vanilla)	25.92	46.01	68.12	67.12	51.79

Appendix

Table 7: Comprehensive performance comparison and ablation studies after integrating CSRS.

Steps	M-Vis.	M-Ver.	M-Vist.	WeMath	Avg.
120	27.97	46.01	67.81	71.77	53.39
125	27.52	47.24	68.06	71.64	53.62
130	28.03	46.18	67.25	70.93	53.10
135	28.01	46.75	67.99	71.62	53.59
140	28.25	47.34	67.63	70.89	53.53

Appendix

Table 8: Performance across different training steps.

Method	M-Vis.	M-Ver.	M-Vist.	WeMath
CSRS (Ran.)	27.63	46.35	67.52	72.01
CSRS (Fix.)	27.97	46.01	67.81	71.77

Appendix

Table 9: Ablation of sampling strategies.

Method	Relative Training Time	Acc.
CSRS Fixed	1.0 $\times$	27.97
CSRS Random	1.13 $\times$	27.63

Appendix

Table 10: Efficiency and Accuracy Comparison.

Figure 6: Case studies in geometric reasoning.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 171 papers

When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning
Mar 22, 2026Zhengxian Wu, Kai Shi, Chuanrui Zhang +10Multimodal Reasoning Self-Training for Language Models
OPPO AI Center · Tsinghua University · Nanyang Technological University +1
A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
Oct 21, 2025Mengqi Li, Lei Zhao, Anthony Man-Cho So +2Self-Training for Language Models Mathematical Reasoning
The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University · The Chinese University of Hong Kong
Confidence-Orchestrated Self-Evolution against Uncertain LLM Feedback
May 27, 2026Bowen Wei, Nan Wang, Yuqing Zhou +2Self-Training for Language Models RL for Language Model Reasoning
George Mason University · Amazon

Mar 22, 2026J/K move · Enter open · S save

When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning

Zhengxian Wu, Kai Shi, Chuanrui Zhang, Zirui Liao+9

OPPO AI Center · Tsinghua University · Nanyang Technological University+1

Multimodal Reasoning Self-Training for Language Models RL for Language Model Reasoning

Open paper AskPDF arXiv

Recent progress in multimodal large language models has led to strong performance on reasoning tasks, but these improvements largely rely on high-quality annotated data or teacher-model distillation, both of which are costly and difficult to scale. To address this, we propose an unsupervised self-evolution training framework for multimodal reasoning that achieves stable performance improvements without using human-annotated answers or external reward models. For each input, we sample multiple reasoning trajectories and jointly model their within group structure. We use the Actor's self-consistency signal as a training prior, and introduce a bounded Judge based modulation to continuously reweight trajectories of different quality. We further model the modulated scores as a group level distribution and convert absolute scores into relative advantages within each group, enabling more robust policy updates. Trained with Group Relative Policy Optimization (GRPO) on unlabeled data, our method consistently improves reasoning performance and generalization on five mathematical reasoning benchmarks, offering a scalable path toward self-evolving multimodal models. The code are available at https://github.com/OPPO-Mente-Lab/LLM-Self-Judge.

Abstract

Language Model Self-Improvement

Explore similar work

When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning

A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning

Confidence-Orchestrated Self-Evolution against Uncertain LLM Feedback