cs.CVOct 20, 2026

VisualEvolve: A Skill Self-Evolution Framework for Visual Reasoning

Authors: Jiawei Wang, Yucun Li, Xueqi Li, Shenghao Wang, Xinquan Gong, Dan Zhang, Zheqi Lv, Yuquan Le, 

Organizations: National University of Singapore · Hunan University · Shenzhen University · Cornell University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Visual tool use enables vision-language models (VLMs) to acquire task-relevant evidence, but its effectiveness can be limited by predefined visual operations and the tool-use strategies learned during training. Representing these operations and strategies as reusable external skills allows them to be refined through self-evolution without updating model parameters. However, skill refinement can introduce performance regressions. Meanwhile, inspecting accumulated task execution trajectories together with skill evolution history may lead to evolution evidence overload. To address these challenges, we propose VisualEvolve, a self-evolving framework that evolves executable visual operations and tool-use strategies for VLMs through a Select--Execute--Evolve--Evaluate cycle. Skill-Library Snapshot Selection uses performance on a fixed validation set disjoint from execution tasks to select an effective historical snapshot for subsequent evolution, limiting the propagation of performance regressions. Progressive Evidence Access (PEA) guides capability-gap diagnosis through a coarse-to-fine hierarchy comprising the Skill-Performance Report, Evolution Log, Trajectory Summaries, and relevant raw trajectory segments. Across four VLM backbones and six evaluation sets from three benchmarks, VisualEvolve improves over the Initial Skill Library in 23 of 24 settings, with a maximum gain of 15.1 percentage points on VisualProbe Hard using Qwen3.5-Plus. Ablation studies support the contributions of Snapshot Selection and PEA, while further experiments show that skills can be improved using experience from different data settings and directly reused across backbones. Moreover, VisualEvolve enables effective skill self-evolution with over an order of magnitude fewer task-solving trajectories than estimated for representative RL-based methods.

Figures & tables

Figure 1: VisualEvolve versus training-based agents on six evaluation sets. VisualEvolve uses our framework with four backbones; References are agents trained with SFT and/or RL. The dashed circle marks the highest reported reference accuracy in each set. VisualEvolve: Acc@1; †\dagger : Avg@32; ‡\ddagger : Avg@8.
Figure 2: The VisualEvolve framework. VisualEvolve evolves a skill library through multi-round Select–Execute–Evolve–Evaluate cycles.
BackboneMethodV * BenchHRBenchVisualProbeMacro Avg.
4K8KEasyMediumHard
Qwen2.5-VL-7BPixel Reasoner R86.3 †74.0 ‡66.9 ‡58.4 †29.6 †28.8 †57.3
DeepEyes R83.3 †73.2 ‡69.5 ‡60.1 †29.8 †35.1 †58.5
Mini-o388.2 †77.5 ‡73.3 ‡67.0 †50.4 †48.0 †67.4
DeepEyesV281.877.973.8––––
Qwen3-VL-32BCodeVision-32B86.284.376.1––––
Table 1: Main results (RQ1). Accuracy (%) on six evaluation sets from the three benchmarks. For training-based methods, results in rows marked with R are taken from the Mini-o3 paper ( Lai et al., 2026 ) , while results for the remaining methods are taken from their respective original papers.
Figure 4
Figure 4: Generalization of VisualEvolve (RQ3). Macro-average Acc@1 (%) across six evaluation sets from the three benchmarks. In (b), VisualEvolve denotes results from backbone-specific evolution, where the skill library is evolved using the corresponding target backbone.
Figure 5: Evolution budget and inference overhead (RQ4). (a) Number of samples processed by the Worker for VisualEvolve versus estimated RL rollout counts for training-based methods ( Hong et al., 2026 ; Zheng et al., 2026 ; Su et al., 2025 ; Guo et al., 2026 ; Lai et al., 2026 ) , shown on a log scale. (b) Mean API cost per evolution round. (c) Added inference cost and absolute accuracy gain over the Initial Skill Library across benchmarks, using Qwen3.5-Plus.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

SettingRoundΔ\Delta Acc. (%)C/IDiagnosisTool-Use StrategyVisual OperationSkill-Loading Sequences
Defaultr0–r1+3.50/12Under-invocation of crop, unstructured observation, and missing answer verification++ {observe, verify}; ∼\sim {crop}–T2: observe →\rightarrow crop (24.5%)
 r1–r2+5.52/12Redundant crop operations, incomplete search coverage, and inconsistent counting and spatial reasoning++ {count, search}; ∼\sim {crop, observe, verify}–T2: observe →\rightarrow crop (12.0%), search →\rightarrow crop (7.5%)
 r2–r16+4.05/128Persistent errors in color recognition, text reading, counting, localization, and candidate disambiguation++ {color, read_text}; ∼\sim {crop, observe, count, search, verify}–T2: observe →\rightarrow crop (20.5%), read_text →\rightarrow crop (13.5%), color →\rightarrow crop (9.5%)
CodeVisionr0–r1+4.012/9Rotated or upside-down inputs caused errors in reading text, charts, and numerical values++ {rotate, measure, verify}; ∼\sim {crop}rotate †T2: crop →\rightarrow verify (16.5%), measure →\rightarrow crop (11.5%), rotate →\rightarrow measure (10.5%) T3: rotate →\rightarrow measure →\rightarrow crop (8.0%), measure →\rightarrow crop →\rightarrow verify (8.0%) T4: rotate →\rightarrow measure →\rightarrow crop →\rightarrow verify (6.5%)
 r1–r2+10.01/13Rotate could not correct mirrored inputs, coordinate and value-reading errors persisted after transformation++ {flip}; ∼\sim {crop, measure, rotate, verify}flip †T2: rotate →\rightarrow flip (18.5%), crop →\rightarrow verify (14.0%), flip →\rightarrow verify (5.0%) T3: rotate →\rightarrow flip →\rightarrow verify (5.0%)
Appendix
Table 3: Skill-evolution dynamics across rounds. ++ and ∼\sim denote added and modified tool-use strategies, respectively, while †\dagger marks a newly added executable script in the Visual Operation column. Δ\Delta Acc. is the absolute difference in validation accuracy (%). C/I denotes the numbers of correct/incorrect Trajectory Summaries inspected by the Self-Evolving Agent, respectively. T2, T3, and T4 denote ordered skill-loading sequences containing two, three, and four skills, respectively; for example, observe →\rightarrow crop denotes a two-skill loading sequence. Parentheses report the fraction of validation examples in which each sequence occurs, with only rates of at least 5% shown.
Figure 6: Validation accuracy on the full CodeVision validation set and its Rotated and Flipped subsets across the first three evolution rounds (%). Full denotes the complete CodeVision validation set, while Rotated and Flipped denote the subsets whose original images contain rotation and flipping transformations, respectively.
Figure 7: Execution-set size sensitivity. Lines show the best-so-far validation accuracy over 16 rounds for execution-set sizes of 4, 8, 16, and 32; scatter points show the corresponding validation accuracy at each round.
MethodEstimationRL trajectories
DeepEyesV2 ( Hong et al., 2026 )60×256×1660\times 256\times 16245,760
DeepEyes ( Zheng et al., 2026 )80×256×1680\times 256\times 16327,680
Pixel Reasoner ( Su et al., 2025 )3×15,402×83\times 15{,}402\times 8369,648
CodeVision-32B ( Guo et al., 2026 )2×40,000×82\times 40{,}000\times 8≈\approx 640,000
Mini-o3 ( Lai et al., 2026 )200×256×16200\times 256\times 16819,200
Appendix
Table 4: Estimated trajectory budgets of training-based methods. Each estimate follows either S×B×GS\times B\times G for step-based training or E×D×GE\times D\times G for epoch-based training.
Figure 8: Case study of visual-operation expansion and composition. The CodeVision-evolved library first loads the measure skill, then recognizes that the chart is both upside down and mirrored. It composes the newly discovered rotate and flip operations to recover a readable chart before extracting the values and computing the final answer.
Figure 9: Case study of iterative local inspection and verification. The evolved library combines read_text guidance with progressively tighter crops. The agent first localizes the relevant bookshelf region, then repeatedly narrows the visual evidence and verifies the target book spine before returning the title.
More in this topic

Tool Use in VLMs

Tool use in vision-language models (VLMs) studies how models call visual, computational, and retrieval tools to improve grounded perception and reasoning.

See all 117 papers

Explore similar work

CardsList
Jun 29, 2026

Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agents

Improving vision-language models (VLMs) on visual reasoning typically requires retraining or hand-designed prompts and tools. We present Dynamo, a training-free framework that adapts a frozen VLM without any weight updates. On a small labeled training subset, the agent inspects its own correct and incorrect attempts and evolves two complementary capabilities: reusable reasoning skills for cognitive bottlenecks, and executable visual tools for perceptual ones. Each generated tool is paired with a skill that specifies when to invoke it, and both capability types accumulate in a persistent library. Across four visual reasoning benchmarks and five VLM backbones, Dynamo improves direct inference on all 20 model--benchmark settings (avg. +5.6 acc). When the tool set is given in advance, the framework learns when to call each tool, and per-step tool choice improves on every tested backbone. Against task-specific RL (VTool-R1, DeepEyes), Dynamo closes 65--99% of the RL gap at a fraction of the compute, and combines additively with RL when available.

Yutao Sun, Yanting Miao, Hao-Xuan Ma +8

1Qwen Large Model Application Team, Alibaba · 2Zhejiang University · University of Waterloo +3

Tool Use in VLMsVLM AdaptationTool-Using Vision-Language AgentsVLM ReasoningVisually Grounded ReasoningVision-Language Agents
AskPDF
May 20, 2026

RISE: Reliable Improvement in Self-Evolving Vision-Language Models

Vision-language models (VLMs) have achieved strong multimodal reasoning capabilities, but further improving them still relies heavily on large-scale human-constructed supervision for post-training. Such supervision is costly to obtain, especially for reasoning-intensive multimodal tasks where questions, answers, and feedback signals must be carefully designed. This motivates self-evolving learning, where a model improves itself through a dual-role closed loop: a questioner autonomously poses questions and a solver learns to solve them. However, we observe that current VLM self-evolving methods still face three major challenges: coarse-grained role alternation delays the interaction between question generation and solver adaptation; generated questions can progressively degrade in quality; and question types may collapse toward a narrow distribution. These issues limit the efficiency and reliability of self-evolution. Thus, we propose RISE, a reliable self-evolving framework for vision-language models. RISE is built on three complementary designs: fine-grained role alternation, which shortens the feedback loop between the questioner and the solver to improve efficiency; a quality supervisor, which improves question validity and pseudo-label reliability; and skill-aware dynamic balancing, which mitigates mode collapse and maintains broad skill coverage during evolution. Together, these components enable more reliable and effective self-evolution from unlabeled images. Experiments on two VLM backbones across seven benchmarks show that RISE consistently improves the base models, yielding broad and sustained gains. Our code is publicly available at https://github.com/AMAP-ML/RISE.

Chaoran Xu, Yingmao Miao, Pengfei Zhang +3

AMAP, Alibaba Group

Vision-Language ModelsVLM AdaptationSelf-TrainingVLM ReasoningLanguage Model Self-Improvement
AskPDF
Jun 11, 2026

Self-Evolving Visual Questioner

Vision-language models (VLMs) are typically trained as passive answerers, while their ability to actively ask diverse, non-trivial, visual-centric and grounded questions remains underexplored. Existing visual questioners' performance is bottlenecked by the availability of high-quality training data or the cost of curating them. We show that a VLM can continuously improve itself as a visual questioner without any external supervision. We propose a self-evolving framework that uses a VLM itself as both a proposer and a filter to produce harder, more informative, and visual-centric questions, while maintaining their exploration diversity to avoid training collapse. These questions are then used to train the VLM in both questioner and answerer modes. To evaluate the questioner, we introduce an agentic protocol that assesses questions along perception, reasoning, and diversity dimensions. Experiments across various backbone VLMs show that our method substantially enhances the quality and substantially expands the difficulty boundary of autonomous question generation. Under the same budget, our self-supervision is more effective than training on the static source data. Moreover, the self-evolving questioner remains a competitive or even better answerer.

Yijun Liang, Hengguang Zhou, Ming Li +3

University of Maryland, College Park · University of California, Los Angeles · Peking University +2

Visual Question AnsweringLanguage Model Self-Improvement
AskPDF