Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning

Kevin Frans; Alexander M Rush; Sergey Levine; Pieter Abbeel; Ashvin Nair

cs.LGNov 3, 2026

Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning

Authors: Kevin Frans, Alexander M Rush, Sergey Levine, Pieter Abbeel, Ashvin Nair

Organizations: Cursor Research · UC Berkeley

Abstract

Naive on-policy RL is often bottlenecked by the capabilities of the base model --- insufficient coverage can result in an inability to learn from certain rewards. At the same time, rewards in modern post-training systems often come from rich textual sources, e.g. judge models with a rubric of criteria. Can we utilize such structures to provide a rich directional signal that enables improving on otherwise difficult to optimize rewards? In this work, we examine a framework we refer to as targeted guidance that learns from per-turn textual feedback via an on-policy self-distillation objective. Our focus is not only on directly improving hard-to-optimize behaviors, but on how to do so throughout the training process without degrading baseline RL performance. Over a range of settings across an agentic multi-turn software engineering environment, we find success by combining targeted self-distillation with a per-criteria RL reward, and by limiting feedback to turns where the policy specifically fails. We further examine the design choices that lead to stable entropy and reduce oscillatory behavior throughout training.

Figures & tables

Figure 1: Naive RL fails to optimize criterion that fall outside the support of base policies. Exploration issues in RL can manifest as slow learning (middle) or in the worst case a complete failure to improve (left). On the other hand, targeted guidance enables policies to directionally improve specific aspects of their behavior and thus continue to learn.

Figure 2: Targeted guidance derives per-token signal from per-turn hints. During the course of a standard RL procedure, in cases where rollouts fail to satisfy certain criteria, e.g. to utilize a certain writing style, textual feedback is employed to guide the policy towards the solution. We instantiate this via a per-token KL loss against a feedback-conditioned expert policy.

Figure 3: Didactic example: Targeted guidance teaches new tool call formats. In cases where the base policy fails to fulfill certain criterion, e.g. to utilize a certain tool call format, textual feedback is utilized to guide the policy towards the solution. We instantiate this via a per-token KL loss against a feedback-conditioned expert policy. As shown on the right in red , only the token in which the student and expert diverge (in this case, the tool format delimiter) receives a strong signal.

Method	Markdown	Sol. Desc	Req. Order	Approval	Bolded	Cmd Ran	Suggest Work	Pass Rate
Baseline	$0.00_{\pm.00}$	$0.70_{\pm.06}$	$0.04_{\pm.03}$	$0.00_{\pm.00}$	$0.43_{\pm.07}$	$0.00_{\pm.00}$	$0.03_{\pm.03}$	$\mathbf{0.33}_{\pm.05}$
GRPO	$0.00_{\pm.00}$	$\mathbf{0.96}_{\pm.02}$	$0.48_{\pm.08}$	$0.00_{\pm.00}$	$\mathbf{0.95}_{\pm.01}$	$0.00_{\pm.00}$	$0.20_{\pm.07}$	$\mathbf{0.33}_{\pm.07}$
OPSD	$0.77_{\pm.04}$	$0.88_{\pm.03}$	$0.29_{\pm.03}$	$0.25_{\pm.09}$	$0.90_{\pm.02}$	$0.01_{\pm.00}$	$0.44_{\pm.04}$	$0.30_{\pm.05}$
Online SFT	$0.01_{\pm.01}$	$0.78_{\pm.03}$	$0.29_{\pm.03}$	$0.02_{\pm.01}$	$0.69_{\pm.03}$	$0.00_{\pm.00}$	$0.28_{\pm.07}$	$0.26_{\pm.05}$
LaTRO	$0.00_{\pm.00}$	$0.57_{\pm.19}$	$0.03_{\pm.02}$	$0.00_{\pm.00}$	$0.21_{\pm.11}$	$0.00_{\pm.00}$	$0.07_{\pm.04}$	$0.14_{\pm.09}$
Verifree	$0.01_{\pm.02}$	$0.83_{\pm.04}$	$0.26_{\pm.14}$	$0.01_{\pm.01}$	$0.65_{\pm.19}$	$0.00_{\pm.00}$	$0.15_{\pm.11}$	$0.20_{\pm.06}$

Table 1: A comparison of methods that aim to learn a suite of stylistic criteria (main columns), while maintaining primary coding pass-rate (right column). Errors represent standard deviation over the last 32 steps of training.

Figure 4: A haiku-style poem produced by a policy trained via targeted guidance. Feedback is inserted at every turn in which the desired criterion is not achieved.

Figure 5: Targeted guidance and alternatives as used to train models that for each agent turn report status updates as a poem. The baseline GRPO run fails to optimize this out-of-support objective entirely. Naive OPSD learns, but is unstable. A variant of targeted feedback where feedback is always provided (rather than only on failing turns) results in a regression in coding pass-rate.

Figure 6: Entropy spikes can occur if distillation weight is too high. In our setting, this results in a spike in average token length, followed shortly by a collapse in pass-rate. With low enough distillation weight, the policy is stable throughout RL and style rewards can continue to improve without coding pass-rate degradation.

Figure 7: Including thinking tokens in the distillation objective is crucial for stability. In cases where the thinking is excluded from distillation signal, style criterion tend to display oscillatory behavior where certain criterion may improve, but later fall back down.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 633 papers

Self-Distilled Agentic Reinforcement Learning
May 14, 2026Zhengxi Lu, Zhiyuan Yao, Zhuowen Han +8Agentic RL Self-Distillation
Zhejiang University · Meituan · Tsinghua University
Group-Reflective Self-Distillation for Agentic Reinforcement Learning
Jul 30, 2026Binbin Zheng, Zijun Xie, Guanqun Zhao +4Agentic RL Credit Assignment in RL
University of Science and Technology of China · 2Baidu Inc. · 3Peking University +3
Adaptive Supervised Anchoring for On-Policy Self-Distillation
Aug 8, 2026Meilin Yang, Zixuan Ding, Jianhao Nie +5Supervised Fine-Tuning On-Policy Self-Distillation
Renmin University of China, Beijing, China · Renmin University of China, Beijing, China.

May 14, 2026J/K move · Enter open · S save

Self-Distilled Agentic Reinforcement Learning

Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang+7

Zhejiang University · Meituan · Tsinghua University

Agentic RL Self-Distillation On-Policy Distillation RL for LLM Agents

Open paper AskPDF arXiv

Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL--OPSD baselines across model scales.

Abstract

RL for LLM Agents

Explore similar work

Self-Distilled Agentic Reinforcement Learning

Group-Reflective Self-Distillation for Agentic Reinforcement Learning

Adaptive Supervised Anchoring for On-Policy Self-Distillation