Post-training for long-horizon agentic tasks has a tension between compute efficiency and generalization. While supervised fine-tuning (SFT) is compute efficient, it often suffers from out-of-domain (OOD) degradation. Conversely, end-to-end reinforcement learning (E2E RL) preserves OOD capabilities, but incurs high compute costs due to many turns of on-policy rollout. We introduce PivotRL, a novel framework that operates on existing SFT trajectories to combine the compute efficiency of SFT with the OOD accuracy of E2E RL. PivotRL relies on two key mechanisms: first, it executes local, on-policy rollouts and filters for pivots: informative intermediate turns where sampled actions exhibit high variance in outcomes; second, it utilizes rewards for functional-equivalent actions rather than demanding strict string matching with the SFT data demonstration. We theoretically show that these mechanisms incentivize strong learning signals with high natural gradient norm, while maximally preserving policy probability ordering on actions unrelated to training tasks. In comparison to standard SFT on identical data, we demonstrate that PivotRL achieves +4.17% higher in-domain accuracy on average across four agentic domains, and +10.04% higher OOD accuracy in non-agentic tasks. Notably, on agentic coding tasks, PivotRL achieves competitive accuracy with E2E RL with 4x fewer rollout turns. PivotRL is adopted by NVIDIA's Nemotron-3-Super-120B-A12B, acting as the workhorse in production-scale agentic post-training.
Figures & tables
Benchmark
Base
SFT
PivotRL
Δ vs SFT
τ2 -Bench
44.35
58.44
63.81
+5.37
SWE-Bench Verified
19.07
37.40
32.67
-4.73
Terminal-Bench
5.42
13.75
20.00
+6.25
BrowseComp
2.50
1.50
11.30
+9.80
Table 1: In-domain results. Base denotes Qwen3-30B-A3B-Thinking-2507 ; SFT and PivotRL use identical training data. PivotRL outperforms SFT on three of four benchmarks, with an average gain of +14.11 over Base versus +9.94 for SFT.
OOD Benchmark
Base
Δ SFT
Δ PivotRL
IFBench
52.04
-11.46
+0.82
AIME25
86.04
-19.72
-1.20
MATH500
98.05
-8.51
+0.35
LiveCodeBench
66.52
-7.76
-0.17
Scicode
36.83
-11.39
+1.90
MMLU-Pro
80.84
-2.99
+0.31
Table 2: OOD performance change ( Δ ) relative to Base, averaged across the four training runs in Table 1 . SFT degrades by −9.83 on average; PivotRL maintains near-zero change ( +0.21 ).
Benchmark
Base
τ2
SWE
sft
rl
sft
rl
Peak In-Domain
–
58.44 (+14.09)
63.81 (+19.46)
37.40 (+18.33)
32.67 (+13.60)
IFBench
52.04
44.42 (-7.62)
53.29 (+1.25)
38.80 (-13.24)
54.79 (+2.75)
AIME25
86.04
76.25 (-9.79)
85.63 (-0.41)
82.29 (-3.75)
85.00 (-1.04)
MATH500
98.05
98.00 (-0.05)
98.35 (+0.30)
98.30 (+0.25)
98.65 (+0.60)
LiveCodeBench
66.52
64.98 (-1.54)
66.96 (+0.44)
57.27 (-9.25)
66.96 (+0.44)
Table 3: Per-domain OOD breakdown across four single-domain training runs. Each cell shows score followed by Δ relative to Base. PivotRL (rl) consistently preserves OOD performance, whereas SFT (sft) causes broad regression—particularly after terminal-domain training.
Figure 1: SWE-Bench accuracy vs. (a) cumulative rollout turns and (b) cumulative rollout time during training, using the same number of compute nodes. PivotRL reaches comparable accuracy to E2E RL with ∼4× fewer rollout turns and ∼5.5× less wall-clock time.
Configuration
τ2 -Bench
Full PivotRL ( Dadv + functional reward)
63.81
− Pivot filtering ( Dcand + functional reward)
59.68
− Functional reward ( Dcand + strict reward)
57.34
Same-data SFT
58.44
Base
44.35
Table 4: Ablation on τ2 -Bench. Both pivot filtering and functional reward are necessary; removing either degrades accuracy.
Figure 6
Benchmark
Nemotron-3-Super SFT
After PivotRL Stage
τ2 -Bench
48.00
64.00
SWE-Bench Verified
12.87
61.33
Terminal-Bench 1.1 Core
23.33
34.17
BrowseComp
13.03
25.04
Table 5: Nemotron-3-Super agentic benchmark accuracy before and after the RL stage that included PivotRL environments for the agentic verticals and other RL environments for reasoning and chat.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
τ2 -Airline
τ2 -Retail
τ2 -Telecom
τ2 -Average
gpt-oss-120b-high
55.33
58.19
67.20
60.24
gpt-oss-20b-high
43.33
50.20
52.63
48.72
Qwen3-235B-A22B-Thinking-2507
50.00
66.08
46.78
54.29
Qwen3-30B-A3B-Thinking-2507
50.00
54.39
28.65
44.35
Qwen + SFT
51.33
58.19
65.79
58.44
Qwen + Random Pivots ( Dcand )
58.00
63.74
57.31
59.68
Appendix
Table 6: Effect of pivot selection on τ2 -Bench (best checkpoint over training). Random turns improve over the base model and low-reward-mean pivots perform even stronger.
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
Reinforcement learning with verifiable rewards (RLVR) via Group-Relative Policy Optimization (GRPO) is widely used for multi-turn VLM agent training, yet it suffers from zero-gradient silence on uniform failures and coarse episode-level credit assignment. While On-Policy Distillation (OPD) and On-Policy Self-Distillation (OPSD) mitigate sparse rewards using hindsight information, their underlying mechanisms remain poorly understood. Through controlled counterfactual rollback probes across five multi-turn VLM agent benchmarks, we reveal that performance gains in OPSD/OPD are largely driven by physical state rollback at the pivot step, defined as the first unrecoverable action without remaining step budget. However, physical state rollbacks are computationally prohibitive and infeasible in real-world environments. To bridge this gap, we present Pivot-Aware Internalized Visual On-Policy Training (PIVOT), an RL framework that internalizes pivot localization and state restoration directly into token-level parameter updates, eliminating environment rollbacks during RL training and additional skill hints at test time. PIVOT unifies three functional roles within a single architecture: a failure Analyzer non-invasively localizes the pivot step and diagnoses failure modes from visual trajectory collages and action logs; a detached Teacher re-scores failed tokens under this privileged diagnostic context; and a Student optimizes joint GRPO and confidence-gated OPD objectives. At test time, both Teacher and Analyzer branches are stripped. Evaluated on five multi-turn VLM agent tasks across cognitive grid puzzles, 3D embodied control and navigation, and generative reasoning, PIVOT achieves 0.90 overall accuracy on Qwen2.5-VL-3B (+8% over SFT+GRPO baseline and +5% over previous SOTA) and scales to 0.92 on Qwen3-VL-2B (+12% over SFT+GRPO baseline).
Jiazhou Zhou, Hu Zhou, Yucheng Chen +3
AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) · International Digital Economy Academy (IDEA) · The Hong Kong Polytechnic University +2
Agentic reinforcement learning (RL) is commonly used to enhance the capabilities of large language models (LLMs) in solving complex multi-turn tasks. During rollout, agentic RL exhibits a workload pattern distinct from that of traditional RL: multi-turn interactions repeatedly introduce prefill requests, causing prefill and decode workloads to coexist throughout rollout. This workload makes prefill-decode (PD) disaggregation, a design widely used for online LLM serving, promising for speeding up agentic RL rollout. This paper shows that PD disaggregation can improve rollout efficiency by (a) mitigating interference among data-parallel attention modules, (b) enabling stage-specific parallelism, and (c) reducing end-to-end trajectory completion time. However, a fixed prefill-decode allocation may fail to sustain these benefits as workload composition varies across tasks and over time. We therefore present Prorsa, an agentic RL framework for prediction-guided prefill-decode resource allocation. Prorsa observes the lifecycle of every trajectory to forecast short-horizon prefill and decode demand. These forecasts then guide worker allocation decisions and trigger role switching of rollout engines, aligning prefill and decode capacity with anticipated workload demand. Across synchronous and asynchronous agentic RL settings, Prorsa achieves end-to-end rollout speedups of 1.11x-4.05x over prefill-decode colocation.
Chiheng Lou, Xiangwei Deng, Bangjun Xiao +4
†School of Computer Science, Peking University · ‡LLM-Core Xiaomi
Agentic Reinforcement Learning (RL) has empowered Large Language Models (LLMs) to utilize tools like Python interpreters for complex problem-solving. However, for parameter-constrained models (e.g., 4B--7B), the exploration phase is often plagued by frequent execution failures, creating noisy trajectories that hinder policy optimization. Under standard outcome-based reward settings, this noise leads to a critical credit assignment issue, where erroneous actions are inadvertently reinforced alongside successful outcomes. Existing mitigations face a dilemma: dense rewards often trigger reward hacking, while supersampling incurs prohibitive computational costs. To address these challenges, we propose CLEANER. Distinct from external filtering methods, CLEANER exploits the model's intrinsic self-correction capabilities to eliminate error-contaminated context directly during data collection. At its core, the Similarity-Aware Adaptive Rollback (SAAR) mechanism autonomously constructs clean, purified trajectories by retrospectively replacing failures with successful self-corrections. Based on semantic similarity, SAAR adaptively regulates replacement granularity from shallow execution repairs to deep reasoning substitutions. By training on these self-purified paths, the model internalizes correct reasoning patterns rather than error-recovery loops. Empirical results on AIME24/25, GPQA, and LiveCodeBench show average accuracy gains of 6%, 3%, and 5% over baselines. Notably, CLEANER matches state-of-the-art performance using only one-third of the training steps, highlighting trajectory purification as a scalable solution for efficient agentic RL. Our models and code are available at GitHub