Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning

Kevin Frans; Alexander M Rush; Sergey Levine; Pieter Abbeel; Ashvin Nair

cs.LGNov 3, 2026

Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning

Authors: Kevin Frans, Alexander M Rush, Sergey Levine, Pieter Abbeel, Ashvin Nair

Organizations: Cursor Research · UC Berkeley

Abstract

Naive on-policy RL is often bottlenecked by the capabilities of the base model --- insufficient coverage can result in an inability to learn from certain rewards. At the same time, rewards in modern post-training systems often come from rich textual sources, e.g. judge models with a rubric of criteria. Can we utilize such structures to provide a rich directional signal that enables improving on otherwise difficult to optimize rewards? In this work, we examine a framework we refer to as targeted guidance that learns from per-turn textual feedback via an on-policy self-distillation objective. Our focus is not only on directly improving hard-to-optimize behaviors, but on how to do so throughout the training process without degrading baseline RL performance. Over a range of settings across an agentic multi-turn software engineering environment, we find success by combining targeted self-distillation with a per-criteria RL reward, and by limiting feedback to turns where the policy specifically fails. We further examine the design choices that lead to stable entropy and reduce oscillatory behavior throughout training.

Figures & tables

Figure 1: Naive RL fails to optimize criterion that fall outside the support of base policies. Exploration issues in RL can manifest as slow learning (middle) or in the worst case a complete failure to improve (left). On the other hand, targeted guidance enables policies to directionally improve specific aspects of their behavior and thus continue to learn.

Figure 2: Targeted guidance derives per-token signal from per-turn hints. During the course of a standard RL procedure, in cases where rollouts fail to satisfy certain criteria, e.g. to utilize a certain writing style, textual feedback is employed to guide the policy towards the solution. We instantiate this via a per-token KL loss against a feedback-conditioned expert policy.

Figure 3: Didactic example: Targeted guidance teaches new tool call formats. In cases where the base policy fails to fulfill certain criterion, e.g. to utilize a certain tool call format, textual feedback is utilized to guide the policy towards the solution. We instantiate this via a per-token KL loss against a feedback-conditioned expert policy. As shown on the right in red , only the token in which the student and expert diverge (in this case, the tool format delimiter) receives a strong signal.

Method	Markdown	Sol. Desc	Req. Order	Approval	Bolded	Cmd Ran	Suggest Work	Pass Rate
Baseline	$0.00_{\pm.00}$	$0.70_{\pm.06}$	$0.04_{\pm.03}$	$0.00_{\pm.00}$	$0.43_{\pm.07}$	$0.00_{\pm.00}$	$0.03_{\pm.03}$	$\mathbf{0.33}_{\pm.05}$
GRPO	$0.00_{\pm.00}$	$\mathbf{0.96}_{\pm.02}$	$0.48_{\pm.08}$	$0.00_{\pm.00}$	$\mathbf{0.95}_{\pm.01}$	$0.00_{\pm.00}$	$0.20_{\pm.07}$	$\mathbf{0.33}_{\pm.07}$
OPSD	$0.77_{\pm.04}$	$0.88_{\pm.03}$	$0.29_{\pm.03}$	$0.25_{\pm.09}$	$0.90_{\pm.02}$	$0.01_{\pm.00}$	$0.44_{\pm.04}$	$0.30_{\pm.05}$
Online SFT	$0.01_{\pm.01}$	$0.78_{\pm.03}$	$0.29_{\pm.03}$	$0.02_{\pm.01}$	$0.69_{\pm.03}$	$0.00_{\pm.00}$	$0.28_{\pm.07}$	$0.26_{\pm.05}$
LaTRO	$0.00_{\pm.00}$	$0.57_{\pm.19}$	$0.03_{\pm.02}$	$0.00_{\pm.00}$	$0.21_{\pm.11}$	$0.00_{\pm.00}$	$0.07_{\pm.04}$	$0.14_{\pm.09}$
Verifree	$0.01_{\pm.02}$	$0.83_{\pm.04}$	$0.26_{\pm.14}$	$0.01_{\pm.01}$	$0.65_{\pm.19}$	$0.00_{\pm.00}$	$0.15_{\pm.11}$	$0.20_{\pm.06}$

Table 1: A comparison of methods that aim to learn a suite of stylistic criteria (main columns), while maintaining primary coding pass-rate (right column). Errors represent standard deviation over the last 32 steps of training.

Figure 4: A haiku-style poem produced by a policy trained via targeted guidance. Feedback is inserted at every turn in which the desired criterion is not achieved.

Figure 5: Targeted guidance and alternatives as used to train models that for each agent turn report status updates as a poem. The baseline GRPO run fails to optimize this out-of-support objective entirely. Naive OPSD learns, but is unstable. A variant of targeted feedback where feedback is always provided (rather than only on failing turns) results in a regression in coding pass-rate.

Figure 6: Entropy spikes can occur if distillation weight is too high. In our setting, this results in a spike in average token length, followed shortly by a collapse in pass-rate. With low enough distillation weight, the policy is stable throughout RL and style rewards can continue to improve without coding pass-rate degradation.

Figure 7: Including thinking tokens in the distillation objective is crucial for stability. In cases where the thinking is excluded from distillation signal, style criterion tend to display oscillatory behavior where certain criterion may improve, but later fall back down.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 633 papers

May 14, 2026

Self-Distilled Agentic Reinforcement Learning

Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL--OPSD baselines across model scales.

Zhengxi Lu, Zhiyuan Yao, Zhuowen Han +8

Zhejiang University · Meituan · Tsinghua University

Agentic RL Self-Distillation On-Policy Distillation RL for LLM Agents

Ask PDF

Jul 30, 2026

Group-Reflective Self-Distillation for Agentic Reinforcement Learning

Reinforcement learning with verifiable rewards (RLVR) is effective for training large language model agents. However, terminal rewards provide only coarse trajectory-level supervision, leaving successful behaviors, recurring mistakes, and incidental choices entangled in the same outcome signal. Existing agentic self-distillation methods enrich sparse supervision with natural-language skills, but skills retrieved externally or extracted from a single trajectory by stronger models may mismatch current experience, exceed the policy's capability, or remain path-specific. We propose Group-Reflective Self-Distillation (GRSD), which derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts. For each prompt, the policy reflects on each verified trajectory in an on-policy group, and a stop-gradient snapshot contrasts the resulting reflections from successful and failed rollouts to construct group-level privileged guidance. Conditioned on this guidance, a self-teacher refines turn-level credit assignment by modulating outcome-based advantages while preserving the verifier-determined learning direction. Experiments across multiple agentic environments and model scales demonstrate that GRSD consistently outperforms competitive baselines and generalizes more effectively to unseen tasks.

Binbin Zheng, Zijun Xie, Guanqun Zhao +4

University of Science and Technology of China · 2Baidu Inc. · 3Peking University +3

Agentic RL Credit Assignment in RL Reinforcement Learning with Verifiable Rewards Self-Distillation

Ask PDF

Aug 8, 2026

Adaptive Supervised Anchoring for On-Policy Self-Distillation

On-policy self-distillation (OPSD) adapts a language model by distilling guidance from a frozen teacher on trajectories sampled from the student. Its effectiveness, however, depends critically on the quality of those trajectories. We show that when student rollouts drift from target trajectories, conditioning the teacher on off-target prefixes substantially weakens its task-relevant supervision. Controlled prefix-corruption experiments expose this failure mode, which we term rollout-conditioned signal degradation. To address this problem, we propose a unified training framework that separates two complementary supervision pathways. The first retains rollout-conditioned distribution matching, providing guidance on states the student actually visits. The second applies supervised cross-entropy on canonical ground-truth contexts, avoiding the incompatibility of imposing target tokens on erroneous rollout prefixes. Token-level rollout-target alignment is used to adapt the strength of the canonical-context anchor, emphasizing it during cold start and relaxing it as rollout quality improves. Experiments across multiple model scales, two task families, and general-reasoning benchmarks show that the proposed approach improves task acquisition over OPSD while preserving general capabilities, resulting in a more favorable empirical plasticity-stability trade-off. These findings identify context quality as a central bottleneck in on-policy self-distillation and demonstrate the value of separating rollout-conditioned guidance from canonical supervision.

Meilin Yang, Zixuan Ding, Jianhao Nie +5

Renmin University of China, Beijing, China · Renmin University of China, Beijing, China.

Supervised Fine-Tuning On-Policy Self-Distillation Language Model Distillation LLM Reasoning

Ask PDF

Abstract

RL for LLM Agents

Explore similar work

Self-Distilled Agentic Reinforcement Learning

Group-Reflective Self-Distillation for Agentic Reinforcement Learning

Adaptive Supervised Anchoring for On-Policy Self-Distillation