cs.AIFeb 24, 2026
SavePyVision-RL: Forging Open Agentic Vision Models via RL
Organizations: Shanghai AI Lab · UMD · CUHK · THU · Shanda AI Research, Tokyo · Rice University
Abstract
Reinforcement learning for agentic multimodal models often suffers from interaction collapse, where models learn to reduce tool usage and multi-turn reasoning, limiting the benefits of agentic behavior. We introduce PyVision-RL, a reinforcement learning framework for open-weight multimodal models that stabilizes training and sustains interaction. Our approach combines an oversampling-filtering-ranking rollout strategy with an accumulative tool reward to prevent collapse and encourage multi-turn tool use. Using a unified training pipeline, we develop PyVision-Image and PyVision-Video for image and video understanding. For video reasoning, PyVision-Video employs on-demand context construction, selectively sampling task-relevant frames during reasoning to significantly reduce visual token usage. Experiments show strong performance and improved efficiency, demonstrating that sustained interaction and on-demand visual processing are critical for scalable multimodal agents.
Figures & tables
Figure 1: Agentic scaffolds of PyVision -RL. We design two agentic scaffolds for image and video understanding under a unified framework of dynamic tooling with Python. For PyVision -Image, both the system prompt and image hints are injected into the MLLM context, and the images are also loaded into the Python runtime. For PyVision -Video, only the system prompt is injected into the MLLM context, while the video is loaded exclusively into the runtime environment. Given a query, the model interleaves reasoning with executable code blocks ( code_block_0 ) to process multimodal inputs. Execution results ( mm_clue_0 ), including textual outputs and rendered images, are appended to the context and fed back to the model. This interaction loop repeats until a final answer is produced. By restricting video inputs first to the runtime, PyVision -Video enables on-demand context construction, where the agent selectively samples and plots task-relevant frames during reasoning, substantially improving visual token efficiency ( Fig. 2 ).
Figure 2: Comparison between frame sampling and on-demand context construction. (a) Conventional video MLLMs, e.g. , the Qwen-VL series, process videos by uniformly sampling frames and directly injecting them into the model context. (b) In PyVision -Video, we adopt on-demand context construction: the video is loaded only into the Python runtime, and the model selectively samples and plots relevant frames via Python code during the reasoning process, largely improve the token efficiency.
Figure 3: Training dynamics of RL for PyVision -Image. Our training algorithm yields stable optimization and steadily improving performance. Entropy loss and gradient norm decrease smoothly over training, indicating stable RL dynamics. Meanwhile, validation performance on V*, accuracy reward, response length, and the mean number of tool calls consistently increase, showing that the model learns sustained, long-horizon tool-using behavior.
Figure 4: Efficiency performance trade-off on VSI-Bench. Thanks to on-demand context construction, PyVision -Video selectively samples task-relevant frames during reasoning, achieving higher accuracy with substantially fewer visual tokens compared to frame-sampling baselines such as Qwen2.5-VL series.
Figure 5: Ablation of training components. We report the average performance over seven benchmarks (V* avg@32, HRBench-4K, HRBench-8K, MathVision, MathVerse, WeMath, and DynaMath) under different training configurations, each ablating one component of our method. The Ours setting uses a max turn budget of 4, includes the accumulative tool reward, applies standard deviatio sorting for rollout groups, and removes standard deviation normalization term in advantage estimation. All other settings modify exactly one component relative to Ours . Overall, we observe that (1) applying standard deviation sorting or removing standard deviation normalization consistently improves performance, and (2) incorporating the accumulative tool reward or increasing the max turn budget leads to larger performance gains in later training stages. For example, at step 600, a max turn budget of 4 outperforms a budget of 2 by 1.93%.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8:
Figure 9:
Figure 10: Overview of the Oversampling-Filtering-Ranking Framework for Rollout Generation. First, we oversample prompts from the prompt pool, where is the batchsize and is the oversampling parameter. Then, each prompt is sent to rollout worker to generate rollouts, where is the group size in the GRPO-like RL algorithms. In the generated rollouts, some of them are broken. For these rollouts, we give their reward with reward model and calculate each one’s group-level stantard deviation. Based on if it is broken and its group-level standard deviation, we filter and sort these rollouts, and keep top- rollouts as the training samples.
Figure 11: Left : we illustrate the distribution of SFT data of PyVision -Image, containing chart understanding data, from ChartQA, infografic understanding data, from InfoVQA, medical understanding data, from GMAI-Reasoning, math data, from MMK-12, and general VQA data, from LLaVA-CoT and MMPR. Right : we illustrate the RL data distribution of PyVision -Image, containing visual search data, from DeepEyes and Mini-o3, and multi-modal reasoning data, from V-Thinker and WeMath-v2.
Figure 12: Left : we illustrate the distribution of SFT data of PyVision -Video, containing visual spatial reasoning data, from SpaceR, and long video understanding data, from LongVILA. Right : the RL data used in PyVision -Video training is all visual spatial reasoning data, from SpaceR.
Figure 13: Performance Comparison of Different RL Training Settings.
Figure 14: Training dynamics of PyVision -Video’s RL process. Our algorithm makes a stable training and a continuous performance increasing. Entropy loss keeps in a moderate level and grad norm decrease steadily, indicating stable RL optimization. Vilidation score on VSI-Bench-subset, accuracy reward, response length and the average tool call numbers increase steadily during RL, showing that the model learns sustained, long-horizon tool-using behavior. To make validation efficient during training, we sample 400 samples randomly from VSI-Bench as the validation dataset, named as VSI-Bench-subset.
Figure 17: Case Study of Color VQA on TIR-Bench. This task requires PyVision -Image to analyze the pixels in the image. This case illustrates how PyVision -Image handles a color VQA task, which requires pixel-level image processing. PyVision -Image first zooms in on and displays the image, then plots histograms of pixel intensities to examine whether any significant differences exist. The resulting histograms show similar distributions, and based on this pixel-level analysis, PyVision -Image arrives at the correct answer.
Figure 18: Case Study of Rotation Game on TIR-Bench. This task requires PyVision -Image to rotate the images to get the final answers. This case demonstrates how PyVision -Image solves the rotation game task, which requires rotating images to restore them correctly. PyVision -Image initially zooms in on and displays the image, then hypothesizes a candidate rotation angle. It subsequently writes Python code to rotate the image and verifies the result. After confirmation, PyVision -Image produces the correct final answer.
Figure 19: Case Study of Absolute Distance on VSI-Bench. This task requires PyVision -Video to first locate the target object and then estimate its distance or physical size. In this case, PyVision -Video estimates the longest dimension of a table from an indoor video. The model first performs uniform frame sampling to identify views where the table and nearby chairs are clearly visible. Using a standard dining chair as a reference object, PyVision -Video estimates a pixel-to-centimeter scale and converts the table’s pixel span into physical measurements. The table’s horizontal length is found to be significantly larger than its width, while the height is not directly observable and is known to be smaller based on typical furniture proportions. Finally, PyVision -Video esitimates the longest dimension of the table os 270cm.
Figure 20: Case Study of Object Counting on VSI-Bench. This task requires PyVision -Video to count a specific object in a given video. In this case, first, PyVision -Video uniformly samples 15 frames from the video. Then, it identifies 2 different tables in frame 700 and frame 1100. To see if there are additional tables or if the same tables are shown from different angles, the model samples more frames of the video clip between frame 600 to frame 1200. Finally, based on the constructed context, PyVision -Video recognizes two different tables, one wooden table with a vase and chairs and one with a red stand and a TV on top.
Figure 21: Tooling taxonomy distribution of PyVision -Image on versatile benchmarks. On visual search tasks, PyVision -Image almost only use crop tools. On multi-modal reasoning tasks, PyVision -Image significantly use more numerical_analysis tools. On agentic reasoning tasks, i.e., TIR-Bench, PyVision -Image use more diverse tools, including, segmentation , render_marks , etc, and some long-tail operations, showing dynamic tooling’s adaptivability and flexibility.
Figure 22: The distribution of tool using number of PyVision -Image. We plot the tool calling number distribution across different benchmarks and models. Models with a larger max turn budget significantly exhibits more tool calling on all benchmarks. On all benchmarks, PyVision -Image, trained with maximum turn budget as 4, for 700 steps, use more than 3 turns on most samples, presenting the long-horizon tool using ability.
Figure 25: Full tool distribution with no operation. In this figure, we present the full tooling distribution including the no operation as one category, which means the generated Python code just plot the original image without further operation. We find no operation accounts for a large portion, indicating that PyVision -Image repeatedly plot the original image to revisit the visual hint.