Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning

Haozhi Yuan; Handong Li; Yifan Zhang; Qianshan Wei

cs.CVNov 9, 2026

Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning

Authors: Haozhi Yuan, Handong Li, Yifan Zhang, Qianshan Wei

Organizations: Nanjing University · Institute of Automation, Chinese Academy of Sciences

Abstract

Agentic reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) offer complementary routes to post-training. For visual agents, however, tool use presents two related challenges. Models may overlook useful tool-returned observations, while poorly grounded actions can limit the evidence available for subsequent distillation. We introduce Reframe, a post-training framework that treats the observation state, rather than only the output distribution, as an object of alignment. First, Ground-Drop-GRPO (GD-GRPO) trains spatial and temporal specialists with evidence-aware rewards and stochastic removal of the original visual input, encouraging them to acquire grounded evidence and use the returned observations. Second, Observation-Gated-MOPD (OG-MOPD) replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The two stages complement each other, as learning to rely on tool-returned observations can help the policy benefit from observation repair. Neither stage requires a task-specific SFT cold start. Extensive experiments over 14 image and video benchmarks demonstrate that Reframe outperforms both Mix-RL and vanilla MOPD. Ablation results show that GD-GRPO increases observation dependence and tool gain compared with outcome-based GRPO.

Figures & tables

Figure 1: Overview of Reframe: visual agents should produce reliable intermediate results and truly look at them. (a) Outcome-based RL without explicit grounding rewards can permit observation neglect. (b) Vanilla MOPD may supervise trajectories conditioned on a poorly grounded observation state. (c) Our GD-GRPO rewards grounded actions and encourages reliance on tool-returned observations. (d) Our OG-MOPD repairs deficient tool actions before continuing distillation.

Model	Training Properties		Effective Tool Use
Model	Unified Image–Video	No SFT Cold Start	Observation Dependence	Tool Gain
Pixel-Reasoner	$\checkmark$	$\times$	23.25	$+0.97$
Thyme	$\times$	$\times$	34.20	$+1.47$
DeepEyes	$\times$	$\checkmark$	19.47	$+1.32$
DeepEyesV2	$\times$	$\times$	41.93	$-2.45$
Reframe (Ours)	$\checkmark$	$\checkmark$	75.53	$\mathbf{+5.23}$

Table 1: Comparison with representative visual agents. Reframe adopts a predominantly on-policy paradigm to consolidate image and video reasoning within a unified model, while achieving the highest observation dependence and tool gain among the compared models. Observation dependence is measured on V ∗ . Tool gain is measured across seven image benchmarks.

Figure 2: Construction pipeline of Reframe-100K. We first collect public data sources and preprocess them into a raw data pool. We then drop samples answerable from text alone, and audit the rest with 8 rollouts on the full input ( $F$ ) and on the annotated evidence alone ( $E$ ), discarding samples that are (1) too easy or too hard; (2) insufficiently supported by their evidence. Remaining suspicious samples are diagnosed by Qwen and removed only when a final reviewer confirms an answer or evidence error.

Figure 3: Overview of GD-GRPO. The policy samples multi-turn tool-use rollouts. After the first successful tool call, the original input $o_{0}$ is dropped with probability 0.5 while tool-returned observations $o_{t}$ remain visible, so the answer must rely on what the tools have retrieved. Rollouts are grouped by their pre-sampled drop flag $d_{i}$ , and advantages are normalized within each subgroup so that dropout does not confound the comparison.

Figure 4: Overview of OG-MOPD. Tool actions of the student policy are checked against annotated evidence using an IoU threshold. Missing, invalid, or poorly grounded actions are replaced with annotation-derived actions to obtain query-relevant observations. Injected action tokens receive NLL supervision, while retained student-generated tokens are trained with MOPD.

Benchmark	Qwen3.5-9B	Image-RL	Video-RL	Mix-RL	MOPD	Reframe (Ours)
Image Understanding
V ∗	83.77	89.53	86.91	88.48	90.05	91.62 +7.85
HR-Bench-4K	83.13	85.25	82.63	83.25	84.88	84.75 +1.62
HR-Bench-8K	76.88	82.63	81.63	80.50	82.50	83.50 +6.62
VisualProbe	43.88	51.65	51.65	51.46	52.62	57.48 +13.60
ZoomBench	50.53	57.40	56.57	57.51	58.46	59.88 +9.35

Table 2: Comparison of training paradigms. Bold and underline mark the best and second-best scores. Image-RL, Video-RL, and Mix-RL use GD-GRPO; MOPD and Reframe distill the same specialists with vanilla MOPD and OG-MOPD. Video-MME-v2, VSI-Bench, and V-STaR report Non-Lin Score, accuracy/MRA, and

100\times\mathrm{mAM}

, respectively; MLVU reports M-Avg.

Condition	GRPO	GD-GRPO
Image Understanding
Image Avg.	65.85	67.92
Observation Dependence
Natural	86.39	89.53
Direct	85.34	87.43
GT	87.96	93.19

Table 3: GD-GRPO versus outcome-based GRPO for image-specialist training. Image performance and tool gain average seven benchmarks. On V ∗ , observation dependence is

100\times\frac{\mathrm{GT}-(\mathrm{Random}+\mathrm{Blank})/2}{100-(\mathrm{Random}+\mathrm{Blank})/2}

(%).

Benchmark/Condition	Pixel-Reasoner	Thyme	DeepEyes	DeepEyesV2	Reframe (Ours)
Image Understanding
V ∗	81.68	81.68	83.25	81.68	91.62
HR-Bench-4K	72.50	77.13	74.63	71.00	84.75
HR-Bench-8K	65.88	67.88	67.50	61.38	83.50
VisualProbe	34.95	41.75	37.67	33.20	57.48
ZoomBench	40.47	46.51	46.75	41.30	59.88

Table 4: Comparison with existing visual agents. Bold and underline mark the best and second-best averages. Video metrics follow Table 2 . On V ∗ , observation dependence is

100\times\frac{\mathrm{GT}-(\mathrm{Random}+\mathrm{Blank})/2}{100-(\mathrm{Random}+\mathrm{Blank})/2}

(%). Tool gain is averaged over seven image benchmarks.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	GRPO / GD-GRPO	MOPD / OG-MOPD
Initialization	Qwen3.5-9B	Qwen3.5-9B
Trainable parameters	Language model	Language model
Optimizer	Fused AdamW	Fused AdamW
Learning rate	$10^{-6}$	$10^{-6}$
Learning-rate schedule	Cosine	Cosine
Warmup ratio	$0.1$	$0.1$

Appendix

Table 5: Optimization and reward settings. MOPD denotes the shared configuration of vanilla MOPD and OG-MOPD. The observation gate and injected-action loss are enabled only in OG-MOPD. The tool-reward settings apply only to GD-GRPO.

Setting	Image-RL	Video-RL	MOPD / OG-MOPD
Training GPUs	$32$	$32$	$8$
Microbatch per GPU	$4$	$4$	$4$
Gradient accumulation	$8$	$8$	$8$
Rollouts per question	$8$	$8$	$1$
Trajectories per update	$1{,}024$	$1{,}024$	$224$
Questions per update	$128$	$128$	$224$

Appendix

Table 6: Rollout and batch settings. Counts refer to independently sampled trajectories before turn splitting. Repeated optimization of a rollout does not create additional independent samples.

Setting	Benchmark Performance	Observation dependence
Decoding	Greedy	Greedy
Image completion cap	$8{,}192$ per turn	$8{,}192$ per turn
Video completion cap	$16{,}384$ per turn	—
LLM judge	Gemini 3.1 Flash Lite	—

Appendix

Table 7: Evaluation settings.

RL Post-Training

Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.

See all 449 papers

On-Policy Visual Evidence Distillation
Sep 29, 2026Shaohang Wei, Feifan Song, Guangyue Peng +9VLM Distillation Visually Grounded Reasoning
Peking University · CUHK · Nanjing University +1
Reinforcement Learning with Visually Attributable On-Policy Self-Distillation
Oct 19, 2026Shiao Xie, Siyu Chen, Jianwei Lv +2RL for VLMs On-Policy Self-Distillation
Baidu Inc., China
ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents
Jun 30, 2026Binjie Zhang, Mike Zheng ShouReinforcement Learning Group Relative Policy Optimization
Show Lab, National University of Singapore

Sep 29, 2026J/K move · Enter open · S save

On-Policy Visual Evidence Distillation

Shaohang Wei, Feifan Song, Guangyue Peng, Wenhao Yu+8

Peking University · CUHK · Nanjing University+1

VLM Distillation Visually Grounded Reasoning On-Policy Distillation

Open paper AskPDF arXiv

Visual agents solve problems by interleaving reasoning with image operations, and on-policy distillation (OPD) provides guidance from a strong teacher on student-generated interaction trajectories. However, image operations change the evidence available for subsequent reasoning, so local errors in evidence acquisition (Acquire), reading (Read), or answer grounding (Ground) can propagate through the trajectory and lead to incorrect answers. Existing multimodal OPD methods primarily construct or contrast auxiliary views of the original image to strengthen supervision, without explicitly modeling the connections between student actions, resulting observations, and subsequent reasoning. This limits their ability to provide corrections tailored to different failure stages. We introduce Reflection on Visual Evidence (ReVuE), an on-policy distillation method for visual agents. ReVuE compares multiple student-generated trajectories for the same query, summarizes the observed visual evidence, and diagnoses the first failure across the Acquire, Read, and Ground stages. The resulting reflections provide training-time context for the teacher. We group and reweight token-level distillation losses according to how strongly these reflections affect the teacher's predictions. This design translates trajectory-level evidence diagnosis into targeted token-level supervision, guiding students to improve their visual evidence acquisition and reasoning. Across 11 benchmarks spanning the Qwen2.5-VL and InternVL3.5 model families, ReVuE outperforms all evaluated OPD baselines in weighted-average scores for perception, mathematical reasoning, and general tasks. ReVuE also reduces redundancy in reasoning and tool calls while improving tool-call accuracy and task accuracy. Code is available at https://github.com/sylvain-wei/ReVuE

Abstract

RL Post-Training

Explore similar work

On-Policy Visual Evidence Distillation

Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents