Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning

Haozhi Yuan; Handong Li; Yifan Zhang; Qianshan Wei

cs.CVNov 9, 2026

Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning

Authors: Haozhi Yuan, Handong Li, Yifan Zhang, Qianshan Wei

Organizations: Nanjing University · Institute of Automation, Chinese Academy of Sciences

Abstract

Agentic reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) offer complementary routes to post-training. For visual agents, however, tool use presents two related challenges. Models may overlook useful tool-returned observations, while poorly grounded actions can limit the evidence available for subsequent distillation. We introduce Reframe, a post-training framework that treats the observation state, rather than only the output distribution, as an object of alignment. First, Ground-Drop-GRPO (GD-GRPO) trains spatial and temporal specialists with evidence-aware rewards and stochastic removal of the original visual input, encouraging them to acquire grounded evidence and use the returned observations. Second, Observation-Gated-MOPD (OG-MOPD) replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The two stages complement each other, as learning to rely on tool-returned observations can help the policy benefit from observation repair. Neither stage requires a task-specific SFT cold start. Extensive experiments over 14 image and video benchmarks demonstrate that Reframe outperforms both Mix-RL and vanilla MOPD. Ablation results show that GD-GRPO increases observation dependence and tool gain compared with outcome-based GRPO.

Figures & tables

Figure 1: Overview of Reframe: visual agents should produce reliable intermediate results and truly look at them. (a) Outcome-based RL without explicit grounding rewards can permit observation neglect. (b) Vanilla MOPD may supervise trajectories conditioned on a poorly grounded observation state. (c) Our GD-GRPO rewards grounded actions and encourages reliance on tool-returned observations. (d) Our OG-MOPD repairs deficient tool actions before continuing distillation.

Model	Training Properties		Effective Tool Use
Model	Unified Image–Video	No SFT Cold Start	Observation Dependence	Tool Gain
Pixel-Reasoner	$\checkmark$	$\times$	23.25	$+0.97$
Thyme	$\times$	$\times$	34.20	$+1.47$
DeepEyes	$\times$	$\checkmark$	19.47	$+1.32$
DeepEyesV2	$\times$	$\times$	41.93	$-2.45$
Reframe (Ours)	$\checkmark$	$\checkmark$	75.53	$\mathbf{+5.23}$

Table 1: Comparison with representative visual agents. Reframe adopts a predominantly on-policy paradigm to consolidate image and video reasoning within a unified model, while achieving the highest observation dependence and tool gain among the compared models. Observation dependence is measured on V ∗ . Tool gain is measured across seven image benchmarks.

Figure 2: Construction pipeline of Reframe-100K. We first collect public data sources and preprocess them into a raw data pool. We then drop samples answerable from text alone, and audit the rest with 8 rollouts on the full input ( $F$ ) and on the annotated evidence alone ( $E$ ), discarding samples that are (1) too easy or too hard; (2) insufficiently supported by their evidence. Remaining suspicious samples are diagnosed by Qwen and removed only when a final reviewer confirms an answer or evidence error.

Figure 3: Overview of GD-GRPO. The policy samples multi-turn tool-use rollouts. After the first successful tool call, the original input $o_{0}$ is dropped with probability 0.5 while tool-returned observations $o_{t}$ remain visible, so the answer must rely on what the tools have retrieved. Rollouts are grouped by their pre-sampled drop flag $d_{i}$ , and advantages are normalized within each subgroup so that dropout does not confound the comparison.

Figure 4: Overview of OG-MOPD. Tool actions of the student policy are checked against annotated evidence using an IoU threshold. Missing, invalid, or poorly grounded actions are replaced with annotation-derived actions to obtain query-relevant observations. Injected action tokens receive NLL supervision, while retained student-generated tokens are trained with MOPD.

Benchmark	Qwen3.5-9B	Image-RL	Video-RL	Mix-RL	MOPD	Reframe (Ours)
Image Understanding
V ∗	83.77	89.53	86.91	88.48	90.05	91.62 +7.85
HR-Bench-4K	83.13	85.25	82.63	83.25	84.88	84.75 +1.62
HR-Bench-8K	76.88	82.63	81.63	80.50	82.50	83.50 +6.62
VisualProbe	43.88	51.65	51.65	51.46	52.62	57.48 +13.60
ZoomBench	50.53	57.40	56.57	57.51	58.46	59.88 +9.35

Table 2: Comparison of training paradigms. Bold and underline mark the best and second-best scores. Image-RL, Video-RL, and Mix-RL use GD-GRPO; MOPD and Reframe distill the same specialists with vanilla MOPD and OG-MOPD. Video-MME-v2, VSI-Bench, and V-STaR report Non-Lin Score, accuracy/MRA, and

100\times\mathrm{mAM}

, respectively; MLVU reports M-Avg.

Condition	GRPO	GD-GRPO
Image Understanding
Image Avg.	65.85	67.92
Observation Dependence
Natural	86.39	89.53
Direct	85.34	87.43
GT	87.96	93.19

Table 3: GD-GRPO versus outcome-based GRPO for image-specialist training. Image performance and tool gain average seven benchmarks. On V ∗ , observation dependence is

100\times\frac{\mathrm{GT}-(\mathrm{Random}+\mathrm{Blank})/2}{100-(\mathrm{Random}+\mathrm{Blank})/2}

(%).

Benchmark/Condition	Pixel-Reasoner	Thyme	DeepEyes	DeepEyesV2	Reframe (Ours)
Image Understanding
V ∗	81.68	81.68	83.25	81.68	91.62
HR-Bench-4K	72.50	77.13	74.63	71.00	84.75
HR-Bench-8K	65.88	67.88	67.50	61.38	83.50
VisualProbe	34.95	41.75	37.67	33.20	57.48
ZoomBench	40.47	46.51	46.75	41.30	59.88

Table 4: Comparison with existing visual agents. Bold and underline mark the best and second-best averages. Video metrics follow Table 2 . On V ∗ , observation dependence is

100\times\frac{\mathrm{GT}-(\mathrm{Random}+\mathrm{Blank})/2}{100-(\mathrm{Random}+\mathrm{Blank})/2}

(%). Tool gain is averaged over seven image benchmarks.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	GRPO / GD-GRPO	MOPD / OG-MOPD
Initialization	Qwen3.5-9B	Qwen3.5-9B
Trainable parameters	Language model	Language model
Optimizer	Fused AdamW	Fused AdamW
Learning rate	$10^{-6}$	$10^{-6}$
Learning-rate schedule	Cosine	Cosine
Warmup ratio	$0.1$	$0.1$

Appendix

Table 5: Optimization and reward settings. MOPD denotes the shared configuration of vanilla MOPD and OG-MOPD. The observation gate and injected-action loss are enabled only in OG-MOPD. The tool-reward settings apply only to GD-GRPO.

Setting	Image-RL	Video-RL	MOPD / OG-MOPD
Training GPUs	$32$	$32$	$8$
Microbatch per GPU	$4$	$4$	$4$
Gradient accumulation	$8$	$8$	$8$
Rollouts per question	$8$	$8$	$1$
Trajectories per update	$1{,}024$	$1{,}024$	$224$
Questions per update	$128$	$128$	$224$

Appendix

Table 6: Rollout and batch settings. Counts refer to independently sampled trajectories before turn splitting. Repeated optimization of a rollout does not create additional independent samples.

Setting	Benchmark Performance	Observation dependence
Decoding	Greedy	Greedy
Image completion cap	$8{,}192$ per turn	$8{,}192$ per turn
Video completion cap	$16{,}384$ per turn	—
LLM judge	Gemini 3.1 Flash Lite	—

Appendix

Table 7: Evaluation settings.

RL Post-Training

Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.

See all 449 papers

Sep 29, 2026

On-Policy Visual Evidence Distillation

Visual agents solve problems by interleaving reasoning with image operations, and on-policy distillation (OPD) provides guidance from a strong teacher on student-generated interaction trajectories. However, image operations change the evidence available for subsequent reasoning, so local errors in evidence acquisition (Acquire), reading (Read), or answer grounding (Ground) can propagate through the trajectory and lead to incorrect answers. Existing multimodal OPD methods primarily construct or contrast auxiliary views of the original image to strengthen supervision, without explicitly modeling the connections between student actions, resulting observations, and subsequent reasoning. This limits their ability to provide corrections tailored to different failure stages. We introduce Reflection on Visual Evidence (ReVuE), an on-policy distillation method for visual agents. ReVuE compares multiple student-generated trajectories for the same query, summarizes the observed visual evidence, and diagnoses the first failure across the Acquire, Read, and Ground stages. The resulting reflections provide training-time context for the teacher. We group and reweight token-level distillation losses according to how strongly these reflections affect the teacher's predictions. This design translates trajectory-level evidence diagnosis into targeted token-level supervision, guiding students to improve their visual evidence acquisition and reasoning. Across 11 benchmarks spanning the Qwen2.5-VL and InternVL3.5 model families, ReVuE outperforms all evaluated OPD baselines in weighted-average scores for perception, mathematical reasoning, and general tasks. ReVuE also reduces redundancy in reasoning and tool calls while improving tool-call accuracy and task accuracy. Code is available at https://github.com/sylvain-wei/ReVuE

Shaohang Wei, Feifan Song, Guangyue Peng +9

Peking University · CUHK · Nanjing University +1

VLM Distillation Visually Grounded Reasoning On-Policy Distillation

Ask PDF

Oct 19, 2026

Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. However, the resulting corrections may conflate visual evidence with non-visual shifts induced by linguistic priors, response style, or the privileged view itself. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling contaminated corrections may therefore reinforce erroneous token-level updates. To address this issue, we propose reinforcement learning with visually attributable on-policy self-distillation, an evidence-consistent paradigm that augments task-level RLVR supervision with privileged token-level corrections grounded in visual evidence. We instantiate this paradigm with Visual-Attribution Gating (VAG), which uses the verifier-derived advantage as the optimization anchor and validates each privileged correction against the contrast between evidence-present and evidence-degraded teacher views. By admitting only corrections directionally supported by this visual contrast, VAG preserves visually grounded guidance while suppressing inconsistent residuals. Experiments across six visual reasoning benchmarks show that VAG consistently outperforms GRPO and OPSD methods, establishing new state-of-the-art average performance.

Shiao Xie, Siyu Chen, Jianwei Lv +2

Baidu Inc., China

RL for VLMs On-Policy Self-Distillation Reasoning for VLMs VLM Distillation Reinforcement Learning with Verifiable Rewards

Ask PDF

Jun 30, 2026

ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents

Tool-augmented vision-language models (VLMs) can solve multimodal, multi-step tasks by calling external tools, yet they remain fragile in practice. Existing works have two common gaps. Supervised fine-tuning (SFT) is built mostly on successful trajectories and offers little signal for recovery after tool failures, while sparse trajectory-level RL rewards provide limited guidance on which step failed and how to repair it. We introduce ReGRPO (Reflection-augmented Group Relative Policy Optimization), a framework that learns reflection-guided correction in tool-using agents. ReGRPO starts with a structured reflective data engine: we execute near-miss actions to collect grounded failure observations, then build Reflection-of-Thought triplets (ErrorType, Evidence, FixPlan) paired with corrected actions for warm-start SFT. We then optimize reflection tokens and corrective actions jointly within local trajectories using group-relative advantages, and include a reflection-cost term to reduce unnecessary reflection. Experiments on GTA and GAIA show that, under the same backbone and tool suite, ReGRPO consistently outperforms strong open-source baselines and achieves the best results among the compared open-source controllers. Code and RoT data are available at https://github.com/showlab/ReGRPO.

Binjie Zhang, Mike Zheng Shou

Show Lab, National University of Singapore

Reinforcement Learning Group Relative Policy Optimization Tool-Using Vision-Language Agents RL for Tool Use

Ask PDF

Abstract

RL Post-Training

Explore similar work

On-Policy Visual Evidence Distillation

Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents