PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost

Junkeun Yi; Damon Mosk-Aoyama; Baihe Huang; Ritu Gala; Charles Wang; Sugam Dipak Devare; Khushi Bhardwaj; Abhibha Gupta

cs.AIMar 22, 2026

PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost

Authors: Junkeun Yi, Damon Mosk-Aoyama, Baihe Huang, Ritu Gala, Charles Wang, Sugam Dipak Devare, Khushi Bhardwaj, Abhibha Gupta,

Organizations: NVIDIA · UC Berkeley

Abstract

Post-training for long-horizon agentic tasks has a tension between compute efficiency and generalization. While supervised fine-tuning (SFT) is compute efficient, it often suffers from out-of-domain (OOD) degradation. Conversely, end-to-end reinforcement learning (E2E RL) preserves OOD capabilities, but incurs high compute costs due to many turns of on-policy rollout. We introduce PivotRL, a novel framework that operates on existing SFT trajectories to combine the compute efficiency of SFT with the OOD accuracy of E2E RL. PivotRL relies on two key mechanisms: first, it executes local, on-policy rollouts and filters for pivots: informative intermediate turns where sampled actions exhibit high variance in outcomes; second, it utilizes rewards for functional-equivalent actions rather than demanding strict string matching with the SFT data demonstration. We theoretically show that these mechanisms incentivize strong learning signals with high natural gradient norm, while maximally preserving policy probability ordering on actions unrelated to training tasks. In comparison to standard SFT on identical data, we demonstrate that PivotRL achieves +4.17% higher in-domain accuracy on average across four agentic domains, and +10.04% higher OOD accuracy in non-agentic tasks. Notably, on agentic coding tasks, PivotRL achieves competitive accuracy with E2E RL with 4x fewer rollout turns. PivotRL is adopted by NVIDIA's Nemotron-3-Super-120B-A12B, acting as the workhorse in production-scale agentic post-training.

Figures & tables

Benchmark	Base	SFT	PivotRL	$\Delta$ vs SFT
$\tau^{2}$ -Bench	44.35	58.44	63.81	+5.37
SWE-Bench Verified	19.07	37.40	32.67	-4.73
Terminal-Bench	5.42	13.75	20.00	+6.25
BrowseComp	2.50	1.50	11.30	+9.80

Table 1: In-domain results. Base denotes Qwen3-30B-A3B-Thinking-2507 ; SFT and PivotRL use identical training data. PivotRL outperforms SFT on three of four benchmarks, with an average gain of

+14.11

over Base versus

+9.94

for SFT.

OOD Benchmark	Base	$\Delta$ SFT	$\Delta$ PivotRL
IFBench	52.04	-11.46	+0.82
AIME25	86.04	-19.72	-1.20
MATH500	98.05	-8.51	+0.35
LiveCodeBench	66.52	-7.76	-0.17
Scicode	36.83	-11.39	+1.90
MMLU-Pro	80.84	-2.99	+0.31

Table 2: OOD performance change (

\Delta

) relative to Base, averaged across the four training runs in Table 1 . SFT degrades by

-9.83

on average; PivotRL maintains near-zero change (

+0.21

Benchmark	Base	$\tau^{2}$		SWE
Benchmark	Base	sft	rl	sft	rl
Peak In-Domain	–	58.44 (+14.09)	63.81 (+19.46)	37.40 (+18.33)	32.67 (+13.60)
IFBench	52.04	44.42 (-7.62)	53.29 (+1.25)	38.80 (-13.24)	54.79 (+2.75)
AIME25	86.04	76.25 (-9.79)	85.63 (-0.41)	82.29 (-3.75)	85.00 (-1.04)
MATH500	98.05	98.00 (-0.05)	98.35 (+0.30)	98.30 (+0.25)	98.65 (+0.60)
LiveCodeBench	66.52	64.98 (-1.54)	66.96 (+0.44)	57.27 (-9.25)	66.96 (+0.44)

Table 3: Per-domain OOD breakdown across four single-domain training runs. Each cell shows score followed by

\Delta

relative to Base. PivotRL (rl) consistently preserves OOD performance, whereas SFT (sft) causes broad regression—particularly after terminal-domain training.

Figure 1: SWE-Bench accuracy vs. (a) cumulative rollout turns and (b) cumulative rollout time during training, using the same number of compute nodes. PivotRL reaches comparable accuracy to E2E RL with ${\sim}4\times$ fewer rollout turns and ${\sim}5.5\times$ less wall-clock time.

Configuration	$\tau^{2}$ -Bench
Full PivotRL ( $\mathcal{D}_{\mathrm{adv}}$ + functional reward)	63.81
$-$ Pivot filtering ( $\mathcal{D}_{\mathrm{cand}}$ + functional reward)	59.68
$-$ Functional reward ( $\mathcal{D}_{\mathrm{cand}}$ + strict reward)	57.34
Same-data SFT	58.44
Base	44.35

Table 4: Ablation on

\tau^{2}

-Bench. Both pivot filtering and functional reward are necessary; removing either degrades accuracy.

Benchmark	Nemotron-3-Super SFT	After PivotRL Stage
$\tau^{2}$ -Bench	48.00	64.00
SWE-Bench Verified	12.87	61.33
Terminal-Bench 1.1 Core	23.33	34.17
BrowseComp	13.03	25.04

Table 5: Nemotron-3-Super agentic benchmark accuracy before and after the RL stage that included PivotRL environments for the agentic verticals and other RL environments for reasoning and chat.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	$\tau^{2}$ -Airline	$\tau^{2}$ -Retail	$\tau^{2}$ -Telecom	$\tau^{2}$ -Average
gpt-oss-120b-high	55.33	58.19	67.20	60.24
gpt-oss-20b-high	43.33	50.20	52.63	48.72
Qwen3-235B-A22B-Thinking-2507	50.00	66.08	46.78	54.29
Qwen3-30B-A3B-Thinking-2507	50.00	54.39	28.65	44.35
Qwen + SFT	51.33	58.19	65.79	58.44
Qwen + Random Pivots ( $\mathcal{D}_{\mathrm{cand}}$ )	58.00	63.74	57.31	59.68

Appendix

Table 6: Effect of pivot selection on

\tau^{2}

-Bench (best checkpoint over training). Random turns improve over the base model and low-reward-mean pivots perform even stronger.

RL Post-Training

Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.

See all 424 papers

PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents
Sep 28, 2026Jiazhou Zhou, Hu Zhou, Yucheng Chen +3On-Policy Self-Distillation Reinforcement Learning with Verifiable Rewards
AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) · International Digital Economy Academy (IDEA) · The Hong Kong Polytechnic University +2
Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning
Oct 12, 2026Chiheng Lou, Xiangwei Deng, Bangjun Xiao +4Prefill-Decode Disaggregation Agentic RL
†School of Computer Science, Peking University · ‡LLM-Core Xiaomi
CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning
Jan 21, 2026Tianshi Xu, Yuteng Chen, Meng LiAgentic RL Reinforcement Learning
Peking University · NTU, Singapore

Sep 28, 2026J/K move · Enter open · S save

PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents

Jiazhou Zhou, Hu Zhou, Yucheng Chen, Jinyuan Qu+2

AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) · International Digital Economy Academy (IDEA) · The Hong Kong Polytechnic University+2

On-Policy Self-Distillation Reinforcement Learning with Verifiable Rewards RL Post-Training Policy Learning RL for VLMs

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) via Group-Relative Policy Optimization (GRPO) is widely used for multi-turn VLM agent training, yet it suffers from zero-gradient silence on uniform failures and coarse episode-level credit assignment. While On-Policy Distillation (OPD) and On-Policy Self-Distillation (OPSD) mitigate sparse rewards using hindsight information, their underlying mechanisms remain poorly understood. Through controlled counterfactual rollback probes across five multi-turn VLM agent benchmarks, we reveal that performance gains in OPSD/OPD are largely driven by physical state rollback at the pivot step, defined as the first unrecoverable action without remaining step budget. However, physical state rollbacks are computationally prohibitive and infeasible in real-world environments. To bridge this gap, we present Pivot-Aware Internalized Visual On-Policy Training (PIVOT), an RL framework that internalizes pivot localization and state restoration directly into token-level parameter updates, eliminating environment rollbacks during RL training and additional skill hints at test time. PIVOT unifies three functional roles within a single architecture: a failure Analyzer non-invasively localizes the pivot step and diagnoses failure modes from visual trajectory collages and action logs; a detached Teacher re-scores failed tokens under this privileged diagnostic context; and a Student optimizes joint GRPO and confidence-gated OPD objectives. At test time, both Teacher and Analyzer branches are stripped. Evaluated on five multi-turn VLM agent tasks across cognitive grid puzzles, 3D embodied control and navigation, and generative reasoning, PIVOT achieves 0.90 overall accuracy on Qwen2.5-VL-3B (+8% over SFT+GRPO baseline and +5% over previous SOTA) and scales to 0.92 on Qwen3-VL-2B (+12% over SFT+GRPO baseline).

Abstract

RL Post-Training

Explore similar work

PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents

Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning

CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning