PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents

Jiazhou Zhou; Hu Zhou; Yucheng Chen; Jinyuan Qu; Ying-Cong Chen; Lei Zhang

cs.CVSep 28, 2026

PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents

Authors: Jiazhou Zhou, Hu Zhou, Yucheng Chen, Jinyuan Qu, Ying-Cong Chen, Lei Zhang

Organizations: AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) · International Digital Economy Academy (IDEA) · The Hong Kong Polytechnic University · MedVisAI Lab, Lee Kong Chian School of Medicine, Nanyang Technological University, and Centre of AI in Medicine · Tsinghua University

Abstract

Reinforcement learning with verifiable rewards (RLVR) via Group-Relative Policy Optimization (GRPO) is widely used for multi-turn VLM agent training, yet it suffers from zero-gradient silence on uniform failures and coarse episode-level credit assignment. While On-Policy Distillation (OPD) and On-Policy Self-Distillation (OPSD) mitigate sparse rewards using hindsight information, their underlying mechanisms remain poorly understood. Through controlled counterfactual rollback probes across five multi-turn VLM agent benchmarks, we reveal that performance gains in OPSD/OPD are largely driven by physical state rollback at the pivot step, defined as the first unrecoverable action without remaining step budget. However, physical state rollbacks are computationally prohibitive and infeasible in real-world environments. To bridge this gap, we present Pivot-Aware Internalized Visual On-Policy Training (PIVOT), an RL framework that internalizes pivot localization and state restoration directly into token-level parameter updates, eliminating environment rollbacks during RL training and additional skill hints at test time. PIVOT unifies three functional roles within a single architecture: a failure Analyzer non-invasively localizes the pivot step and diagnoses failure modes from visual trajectory collages and action logs; a detached Teacher re-scores failed tokens under this privileged diagnostic context; and a Student optimizes joint GRPO and confidence-gated OPD objectives. At test time, both Teacher and Analyzer branches are stripped. Evaluated on five multi-turn VLM agent tasks across cognitive grid puzzles, 3D embodied control and navigation, and generative reasoning, PIVOT achieves 0.90 overall accuracy on Qwen2.5-VL-3B (+8% over SFT+GRPO baseline and +5% over previous SOTA) and scales to 0.92 on Qwen3-VL-2B (+12% over SFT+GRPO baseline).

Figures & tables

Figure 1: PIVOT Architecture Overview. PIVOT unifies Student, Analyzer, and Teacher roles within a single shared policy $\pi_{\theta}$ . Stage I: SFT cold-starts the Analyzer to predict diagnostic triplets $(t^{\ast},m^{\ast},r^{\ast})$ from visual trajectory collages $C(\tau)$ and action logs. Stage II: For failed rollouts, the Analyzer localizes the pivot step $\hat{t}$ to construct a privileged context $p$ , enabling the detached Teacher to re-score failed tokens under $\tilde{h}_{t}=(p,h_{t})$ for joint GRPO and confidence-gated OPD training. Inference: Analyzer and Teacher branches are stripped, deploying $\pi_{\theta}$ strictly as an unprivileged Student with zero computational and prompt overhead.

Panel of Figure 2 — Figure 2: Analyzer SFT input–output examples. Each row pairs a visual trajectory collage and compact action log with the JSON supervision target. The highlighted log line is the pivot step. Additional environments and samples are in App. H .

Panel of Figure 3 — Figure 3: Hindsight Visual Panels $P_{t^{\ast}}=[o_{t^{\ast}-1},o_{t^{\ast}},o_{t^{\ast}+1}]$ . Localized visual neighborhood extracted around the pivot step $t^{\ast}$ across five benchmark tasks (boundary frames zero-padded).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\mathcal{M}$	Partially observable Markov decision process.
$\mathcal{S},\mathcal{O},\mathcal{A}$	Latent-state, observation, and action spaces.
$\mathcal{P},\Omega$	State-transition and observation kernels.
$\mathcal{R},\gamma$	Per-step reward function and discount factor.
$q,\mathcal{Q}$	Task prompt and task-prompt distribution.
$s_{t},o_{t},a_{t},r_{t}$	Latent state, observation, action, and reward at turn $t$ .

Appendix

Table 3: Notation (Sec. 2 – 3.2 ).

Environment	Pivot $t^{\ast}$	Failure modes $m^{\ast}$
Sokoban	Eq. eq. 2 ; $\mathsf{Feas}$ = box-pushing plan $\leq k$	timeout , deadlock
FrozenLake	Eq. eq. 2 ; $\mathsf{Feas}$ = hole-free path $\leq k$	timeout , deadlock
Navigation	earliest terminal event; else Eq. eq. 2 (travel-time proxy)	format , blocked , stagnation , budget
PrimitiveSkill	Eq. eq. 2 ; unfinished stages vs. remaining turns	timeout , format , wrong_target , order_error
SVG	index of last scored canvas	format , regression , stagnation , mismatch

Appendix

Table 4: Offline pivot certificate

(t^{\ast},m^{\ast})

by environment (Eq. eq. 2 when applicable; Eq. eq. 9 for Sokoban/FrozenLake only).

Table 6

Table 7

Panel of Figure 6 — Figure 6: Sokoban. SFT-stage input–output examples for this environment. Columns 1–2 are the visual trajectory collage and compact action log; column 3 is the JSON supervision target.

Panel of Figure 7 — Figure 7: FrozenLake. SFT-stage input–output examples for this environment. Columns 1–2 are the visual trajectory collage and compact action log; column 3 is the JSON supervision target.

Panel of Figure 8 — Figure 8: PrimitiveSkill. SFT-stage input–output examples for this environment. Columns 1–2 are the visual trajectory collage and compact action log; column 3 is the JSON supervision target.

Panel of Figure 9 — Figure 9: SVG. SFT-stage input–output examples for this environment. Columns 1–2 are the visual trajectory collage and compact action log; column 3 is the JSON supervision target.

Item	Value
Student $\pi_{\theta}$ / teacher $\pi_{\phi}$	Qwen2.5-VL-3B / 7B-Instruct (frozen)
Skill	seed_visual , episode_only ; all frames, no collage
Distillation	all valid tokens; $\lambda_{\mathrm{sdar}}{=}0.01$ , $\beta_{\mathrm{sdar}}{=}5$
RL (shared with PIVOT )	$N{=}8$ , LR $1{\times}10^{-6}$ , KL $0.01$ , invalid pen. $0.1$
Training	batch $16$ /val $128$ ; $250$ updates (Sok. $350$ ); history $2$
Pivot Analyzer / $P_{\hat{t}}$	none

Appendix

Table 13: SDAR settings (3B student, frozen 7B teacher).

Explore similar work

PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents
Sep 30, 2026Yinghui He, Yapei Chang, Khushi Bhardwaj +4Student-Generated Trajectories
Princeton University · NVIDIA · University of Maryland
Rethinking Reinforcement Fine-Tuning in LVLM: Convergence, Reward Decomposition, and Generalization
Apr 21, 2026Carter Adams, Rafael Oliveira, Gabriel Almeida +1Reinforcement Learning With Verifiable Reward Reinforcement Fine-Tuning
Federal University of Bahia
Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry
Jul 4, 2026Weiyang Guo, Zesheng Shi, Longhui Zhang +3Retrying Offline Reinforcement Learning
Harbin Institute of Technology, Shenzhen, China

Sep 30, 2026, cs.AIJ/K move · Enter open · S save

PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents

Yinghui He, Yapei Chang, Khushi Bhardwaj, Daniele Molinari+3

Princeton University · NVIDIA · University of Maryland

Student-Generated Trajectories

Open paper AskPDF arXiv

On-policy distillation (OPD) is a promising approach for training language agents, providing dense teacher supervision on student-generated trajectories. However, in multi-turn interaction, an incorrect action changes the states the student encounters later, so errors compound across turns. In preliminary experiments across three Qwen3 models (8B to 235B), we find that more than half of the failed rollouts contain a pivotal mistake, an action that moves the agent farther from completing the task, and this mistake typically occurs early. These pivotal mistakes often remain recoverable: guiding the model for only a few turns after the pivotal turn can restore task success. We therefore propose PivotOPD, an on-policy distillation framework that jointly trains the student to prevent pivotal mistakes and to recover from the states they create. At each pivotal mistake, a teacher model provides a gold action and then names a recovery action at each of the next few turns. Preventive distillation uses the gold action with reverse KL to steer the student away from the pivotal mistake, while recovery distillation uses the recovery actions with forward KL to transfer recovery behaviors that the student rarely samples. Against 13 baselines on ALFWorld, WebShop, and Search-based QA, PivotOPD achieves the strongest average performance for both Qwen3-1.7B and Qwen3-8B students, improving over the strongest baseline on ALFWorld by +5.5% with the 1.7B student. The gains also transfer to another model family on the software engineering domain, where PivotOPD raises the resolve rate of a Nemotron-3.5 student on SWE-Bench Verified by +3.2%. Project page: https://research.nvidia.com/labs/lpr/pivotopd/

Abstract

Explore similar work

PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents

Rethinking Reinforcement Fine-Tuning in LVLM: Convergence, Reward Decomposition, and Generalization

Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry