Reinforcement learning for agentic multimodal models often suffers from interaction collapse, where models learn to reduce tool usage and multi-turn reasoning, limiting the benefits of agentic behavior. We introduce PyVision-RL, a reinforcement learning framework for open-weight multimodal models that stabilizes training and sustains interaction. Our approach combines an oversampling-filtering-ranking rollout strategy with an accumulative tool reward to prevent collapse and encourage multi-turn tool use. Using a unified training pipeline, we develop PyVision-Image and PyVision-Video for image and video understanding. For video reasoning, PyVision-Video employs on-demand context construction, selectively sampling task-relevant frames during reasoning to significantly reduce visual token usage. Experiments show strong performance and improved efficiency, demonstrating that sustained interaction and on-demand visual processing are critical for scalable multimodal agents.
Figures & tables
Figure 1: Agentic scaffolds of PyVision -RL. We design two agentic scaffolds for image and video understanding under a unified framework of dynamic tooling with Python. For PyVision -Image, both the system prompt and image hints are injected into the MLLM context, and the images are also loaded into the Python runtime. For PyVision -Video, only the system prompt is injected into the MLLM context, while the video is loaded exclusively into the runtime environment. Given a query, the model interleaves reasoning with executable code blocks ( code_block_0 ) to process multimodal inputs. Execution results ( mm_clue_0 ), including textual outputs and rendered images, are appended to the context and fed back to the model. This interaction loop repeats until a final answer is produced. By restricting video inputs first to the runtime, PyVision -Video enables on-demand context construction, where the agent selectively samples and plots task-relevant frames during reasoning, substantially improving visual token efficiency ( Fig. 2 ).
Figure 2: Comparison between frame sampling and on-demand context construction. (a) Conventional video MLLMs, e.g. , the Qwen-VL series, process videos by uniformly sampling frames and directly injecting them into the model context. (b) In PyVision -Video, we adopt on-demand context construction: the video is loaded only into the Python runtime, and the model selectively samples and plots relevant frames via Python code during the reasoning process, largely improve the token efficiency.
Figure 3: Training dynamics of RL for PyVision -Image. Our training algorithm yields stable optimization and steadily improving performance. Entropy loss and gradient norm decrease smoothly over training, indicating stable RL dynamics. Meanwhile, validation performance on V*, accuracy reward, response length, and the mean number of tool calls consistently increase, showing that the model learns sustained, long-horizon tool-using behavior.
Figure 4: Efficiency performance trade-off on VSI-Bench. Thanks to on-demand context construction, PyVision -Video selectively samples task-relevant frames during reasoning, achieving higher accuracy with substantially fewer visual tokens compared to frame-sampling baselines such as Qwen2.5-VL series.
Figure 5: Ablation of training components. We report the average performance over seven benchmarks (V* avg@32, HRBench-4K, HRBench-8K, MathVision, MathVerse, WeMath, and DynaMath) under different training configurations, each ablating one component of our method. The Ours setting uses a max turn budget of 4, includes the accumulative tool reward, applies standard deviatio sorting for rollout groups, and removes standard deviation normalization term in advantage estimation. All other settings modify exactly one component relative to Ours . Overall, we observe that (1) applying standard deviation sorting or removing standard deviation normalization consistently improves performance, and (2) incorporating the accumulative tool reward or increasing the max turn budget leads to larger performance gains in later training stages. For example, at step 600, a max turn budget of 4 outperforms a budget of 2 by 1.93%.
Figure 6
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8:
Figure 9:
Figure 10: Overview of the Oversampling-Filtering-Ranking Framework for Rollout Generation. First, we oversample α∗B prompts from the prompt pool, where B is the batchsize and α is the oversampling parameter. Then, each prompt is sent to rollout worker to generate G rollouts, where G is the group size in the GRPO-like RL algorithms. In the generated rollouts, some of them are broken. For these α∗B∗G rollouts, we give their reward with reward model and calculate each one’s group-level stantard deviation. Based on if it is broken and its group-level standard deviation, we filter and sort these rollouts, and keep top- B∗G rollouts as the training samples.
Figure 11: Left : we illustrate the distribution of SFT data of PyVision -Image, containing chart understanding data, from ChartQA, infografic understanding data, from InfoVQA, medical understanding data, from GMAI-Reasoning, math data, from MMK-12, and general VQA data, from LLaVA-CoT and MMPR. Right : we illustrate the RL data distribution of PyVision -Image, containing visual search data, from DeepEyes and Mini-o3, and multi-modal reasoning data, from V-Thinker and WeMath-v2.
Figure 12: Left : we illustrate the distribution of SFT data of PyVision -Video, containing visual spatial reasoning data, from SpaceR, and long video understanding data, from LongVILA. Right : the RL data used in PyVision -Video training is all visual spatial reasoning data, from SpaceR.
Figure 13: Performance Comparison of Different RL Training Settings.
Figure 14: Training dynamics of PyVision -Video’s RL process. Our algorithm makes a stable training and a continuous performance increasing. Entropy loss keeps in a moderate level and grad norm decrease steadily, indicating stable RL optimization. Vilidation score on VSI-Bench-subset, accuracy reward, response length and the average tool call numbers increase steadily during RL, showing that the model learns sustained, long-horizon tool-using behavior. To make validation efficient during training, we sample 400 samples randomly from VSI-Bench as the validation dataset, named as VSI-Bench-subset.
Figure 14
Figure 17: Case Study of Color VQA on TIR-Bench. This task requires PyVision -Image to analyze the pixels in the image. This case illustrates how PyVision -Image handles a color VQA task, which requires pixel-level image processing. PyVision -Image first zooms in on and displays the image, then plots histograms of pixel intensities to examine whether any significant differences exist. The resulting histograms show similar distributions, and based on this pixel-level analysis, PyVision -Image arrives at the correct answer.
Figure 18: Case Study of Rotation Game on TIR-Bench. This task requires PyVision -Image to rotate the images to get the final answers. This case demonstrates how PyVision -Image solves the rotation game task, which requires rotating images to restore them correctly. PyVision -Image initially zooms in on and displays the image, then hypothesizes a candidate rotation angle. It subsequently writes Python code to rotate the image and verifies the result. After confirmation, PyVision -Image produces the correct final answer.
Figure 19: Case Study of Absolute Distance on VSI-Bench. This task requires PyVision -Video to first locate the target object and then estimate its distance or physical size. In this case, PyVision -Video estimates the longest dimension of a table from an indoor video. The model first performs uniform frame sampling to identify views where the table and nearby chairs are clearly visible. Using a standard dining chair as a reference object, PyVision -Video estimates a pixel-to-centimeter scale and converts the table’s pixel span into physical measurements. The table’s horizontal length is found to be significantly larger than its width, while the height is not directly observable and is known to be smaller based on typical furniture proportions. Finally, PyVision -Video esitimates the longest dimension of the table os 270cm.
Figure 20: Case Study of Object Counting on VSI-Bench. This task requires PyVision -Video to count a specific object in a given video. In this case, first, PyVision -Video uniformly samples 15 frames from the video. Then, it identifies 2 different tables in frame 700 and frame 1100. To see if there are additional tables or if the same tables are shown from different angles, the model samples more frames of the video clip between frame 600 to frame 1200. Finally, based on the constructed context, PyVision -Video recognizes two different tables, one wooden table with a vase and chairs and one with a red stand and a TV on top.
Figure 21: Tooling taxonomy distribution of PyVision -Image on versatile benchmarks. On visual search tasks, PyVision -Image almost only use crop tools. On multi-modal reasoning tasks, PyVision -Image significantly use more numerical_analysis tools. On agentic reasoning tasks, i.e., TIR-Bench, PyVision -Image use more diverse tools, including, segmentation , render_marks , etc, and some long-tail operations, showing dynamic tooling’s adaptivability and flexibility.
Figure 22: The distribution of tool using number of PyVision -Image. We plot the tool calling number distribution across different benchmarks and models. Models with a larger max turn budget significantly exhibits more tool calling on all benchmarks. On all benchmarks, PyVision -Image, trained with maximum turn budget as 4, for 700 steps, use more than 3 turns on most samples, presenting the long-horizon tool using ability.
Figure 21
Figure 25: Full tool distribution with no operation. In this figure, we present the full tooling distribution including the no operation as one category, which means the generated Python code just plot the original image without further operation. We find no operation accounts for a large portion, indicating that PyVision -Image repeatedly plot the original image to revisit the visual hint.
Recent progress in foundation models has shifted toward agentic behavior involving multi-step reasoning and tool use. However, open-source efforts largely focus on text-dominant settings, leaving long-horizon multimodal tasks underexplored. This gap is evident in video tasks requiring sustained temporal understanding and iterative interaction. We present InternVideo3, a framework enhancing these capabilities via Multimodal Contextual Reasoning (MCR). MCR treats understanding as a closed-loop process over a shared, evolving context containing observations, instructions, reasoning, tool actions, and memory. This frames long-video understanding as evidence accumulation and verification. To ensure efficiency, we introduce Multimodal Multi-head Latent Attention (M^2LA), a token-preserving reparameterization compressing KV-cache states while retaining the full token stream. Our staged training includes continued pretraining, short-to-long supervised fine-tuning, rule-based reinforcement learning, and on-policy distillation. Experiments show InternVideo3 achieves strong performance on benchmarks like Video-MME, MLVU, and EgoSchema. We further instantiate the model as a video agent with retrieval tools, demonstrating robust evidence-grounded behavior. Our results suggest that efficient context handling and closed-loop reasoning are vital for adapting open multimodal models toward long-horizon visually grounded agency.
Ziang Yan, Sheng Xia, Jiashuo Yu +10
Shanghai AI Laboratory · Shanghai Innovation Institute · Nanjing University
Recent advancements in Multimodal Large Language Models (MLLMs) have incentivized models to ``think with images'' by actively invoking visual tools during multi-turn reasoning. The common Reinforcement Learning (RL) practice of relying on outcome-based rewards ignores the fact that textual plausibility often masks executive failure, meaning that models may exhibit intuitive textual reasoning while executing imprecise or irrelevant visual actions within their agentic reasoning trajectories. This reasoning-action discrepancy introduces noise that accumulates throughout the multi-turn reasoning process, severely degrading the model's multimodal reasoning capabilities and potentially leading to training collapse. In this paper, we introduce Multimodal Agentic Policy Optimization (MAPO), bridging the gap between textual reasoning and visual actions generated by models within their Multimodal Chain-of-Thought (MCoT). Specifically, MAPO mandates the model to generate explicit textual descriptions for the visual content obtained via tool usage. We then employ a novel advantage estimation that couples the semantic alignment between these descriptions and the actual observations with the task reward. Theoretical findings are provided to justify the rationale behind MAPO, which inherently reduces the variance of gradients, and extensive experiments demonstrate that our method achieves superior performance across multiple visual reasoning benchmarks.
Wenhao Yang, Yu Xia, Jinlong Huang +10
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · AI Business, Alibaba Group +5
Agentic multimodal reasoning extends passive image understanding by allowing VLMs to actively acquire and refine visual evidence through visual tool interactions. Effective visual tool use requires three capabilities: grounding tool calls in visual context, composing tools across multiple steps, and adapting reasoning to tool-returned observations. However, existing approaches largely focus on grounding within fixed tool spaces and rigid invocation patterns, leaving composition and adaptation insufficiently addressed. We present VC-Tooler, which learns visual tool use as a compositional and adaptive capability. To this end, we first build a trajectory bank through a hierarchical synthesis pipeline covering three capability levels: single-tool grounding, multi-tool composition, and diverse tool contexts and interfaces. We then train the model in two stages: a supervised cold start that establishes these capabilities, followed by reinforcement learning that encourages accurate, efficient, and context-aware visual tool use. VC-Tooler achieves state-of-the-art performance among open-source models on both general-purpose and agentic benchmarks, including 95.8% on V* and 35.3% on VTC-Bench, and shows promising transfer under richer tool settings at inference time. Project page: https://w1zheng.github.io/VC-Tooler