Co-Evolution of Policy and Internal Reward for Language Agents

Xinyu Wang; Hanwei Wu; Jingwei Song; Shuyuan Zhang; Jiayi Zhang; Fanqi Kong; Tung Sum Thomas Kwok; Xiao-Wen Chang

cs.LGApr 3, 2026

Co-Evolution of Policy and Internal Reward for Language Agents

Authors: Xinyu Wang, Hanwei Wu, Jingwei Song, Shuyuan Zhang, Jiayi Zhang, Fanqi Kong, Tung Sum Thomas Kwok, Xiao-Wen Chang,

Organizations: McGill University · McMaster University · The University of Hong Kong · Mila ·

Abstract

Large language model (LLM) agents learn by interacting with environments, but long-horizon training remains fundamentally bottlenecked by sparse and delayed rewards. Existing methods typically address this challenge through post-hoc credit assignment or external reward models, which provide limited guidance at inference time and often separate reward improvement from policy improvement. We propose Self-Guide, a self-generated internal reward for language agents that supports both inference-time guidance and training-time supervision. Specifically, the agent uses Self-Guide as a short self-guidance signal to steer the next action during inference, and converts the same signal into step-level internal reward for denser policy optimization during training. This creates a co-evolving loop: better policy produces better guidance, and better guidance further improves policy as internal reward. Across three agent benchmarks, inference-time self-guidance already yields clear gains, while jointly evolving policy and internal reward with GRPO brings further improvements (8%) over baselines trained solely with environment reward. Overall, our results suggest that language agents can improve not only by collecting more experience, but also by learning to generate and refine their own internal reward during acting and learning.

Figures & tables

Figure 1: Sparse environment reward alone may fail to distinguish trajectories with different intermediate quality, leaving GRPO with weakly separated training signals. Our method interleaves self-guided steps with action steps during trajectory generation, so that the same signal can guide action selection at inference time and be aggregated as internal reward at training time. This yields denser and more discriminative trajectory-level supervision for policy optimization.

Figure 2: Comparison between baseline GRPO and GRPO with Self-Guide. Baseline GRPO optimizes a policy using sparse trajectory-level environment rewards. Our method augments each step with a verbal self-guidance signal

z_{t}

: the model first generates

z_{t}

to assess the current trajectory, and then produces action

a_{t}

conditioned on

z_{t}

. The same self-guidance signals are mapped to step-level internal rewards, aggregated into

R_{\mathrm{sg}}(\tau)

, and combined with

R_{\mathrm{env}}(\tau)

via a stage-dependent coefficient

\lambda(u)

for joint policy optimization.

Panel of Table 1 — Table 1: Main results on interactive benchmarks across different models. For ALFWorld, we report the success rate for each task type and the overall result. Delta annotations indicate the absolute gain relative to the corresponding baseline: ReAct for prompting methods and GRPO for RL-based methods. The prompting/RL variant ReAct w/ SG and GRPO w/ SG is highlighted in light gray, and the full RL variant ( GRPO w/ SG & GR ) is highlighted in dark gray ( SG = Self-Guidance, and GR = Guidance Reward ). For ALFWorld, delta annotations are shown only on the All column for readability.

Figure 3: Self-guidance without training already improves performance in structured environments (ALFWorld) but yields inconsistent gains in more complex ones (WebShop), indicating that self-guidance quality depends on task familiarity.

Panel of Figure 4 — Figure 4: Training curves on the three environments with Qwen3-1.7B as base model.

Panel of Figure 5 — Figure 5: Ablation on stage-wise guidance-reward scheduling. Left: Training curves under different reward schedules. Right: Final validation success rates.

Figure 6: Offline self-guidance distillation versus online co-evolution. Although the student improves under offline distillation (a), the distilled model does not transfer reliably to downstream RL: using it as guidance reward (b) or for inference-time guidance only (c) yields unstable or unsustained gains, supporting the need for online co-evolution.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Self-guidance prompt template, applied identically across ALFWorld, SciWorld, and WebShop.

Panel of Figure 8 — Figure 8: Actor prompt template for ALFWorld. Admissible actions and the self-guidance are injected at each step.

Panel of Figure 9 — Figure 9: Actor prompt template for SciWorld. Because the number of admissible actions can exceed 100 per step, the prompt enumerates only the action format grammar rather than the full per-step action list.

Panel of Figure 10 — Figure 10: Actor prompt template for WebShop, adapted from GiGPO ( Feng et al., 2025b ) . Admissible actions and the self-guidance score are injected at each step.

Category	Criterion (ALFWorld & ScienceWorld)
Looping	Any action repeated $\geq\!5$ consecutively, or single action $\geq\!50\%$ of all steps
Redundant Exploration	$\geq\!8$ redundant go to revisits, or same object examined $\geq\!5$ times
Wrong-Object Focus	Irrelevant object examined $\geq\!4$ times, or $\geq\!7$ pick-ups of non-target objects

Appendix

Table 2: Error classification criteria for ALFWorld and ScienceWorld.

Category	Criterion (WebShop)
Query Looping	Same search query issued $\geq\!3$ times (exact or normalized)
Navigation Cycling	back to search clicked $\geq\!3$ times without completing a purchase
Premature Purchase	Agent clicks buy now without selecting any required attribute

Appendix

Table 3: Error classification criteria for WebShop.

Panel of Figure 11 — Figure 11: Error distribution among failed episodes across three benchmarks and three base models. Each bar shows the percentage of failed episodes exhibiting the corresponding error type; episodes may be counted in multiple categories.

Figure 12: Training curves of DAPO and DAPO w/ SG & GR. Our method achieves a final success rate of 78.1% vs. 72.2% for the DAPO baseline.

Panel of Figure 13 — Figure 13: Prompt template used for offline self-guidance model distillation.

Self-Guidance Model	Score	Success Rate (%)
checkpoint 10	39.8	18.0
checkpoint 40	51.2	25.0
checkpoint 80	60.5	27.0

Appendix

Table 4: Effect of self-guidance model checkpoints on a fixed policy. The policy is fixed to ckpt40 , while the self-guidance model is varied across training checkpoints.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 608 papers

NLPG: Natural-Language Policy Gradients for Self-Evolving Language Agents
Sep 27, 2026Xu Liu, WenZhang Wei, Jun Cao +4Continual Learning for LLM Agents Policy Optimization
School of Electronic Information, Wuhan University, Wuhan 430079, China · School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China · State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China
LangMARL: Natural Language Multi-Agent Reinforcement Learning
Apr 1, 2026Huaiyuan Yao, Longchao Da, Xiaoou Liu +3RL for LLM Agents Multi-Agent Reinforcement Learning
Arizona State University · Cisco Research · University of North Carolina at Chapel Hill
Learning Agentic Policy from Action Guidance
May 12, 2026Yuxiang Ji, Zengbin Wang, Yong Wang +6Agentic RL Reinforcement Learning
1Xiamen University · 2AMAP, Alibaba Group · Work done during internship at AMAP, Alibaba Group +1

Sep 27, 2026J/K move · Enter open · S save

NLPG: Natural-Language Policy Gradients for Self-Evolving Language Agents

Xu Liu, WenZhang Wei, Jun Cao, Dehua Peng+3

School of Electronic Information, Wuhan University, Wuhan 430079, China · School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China · State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China

Continual Learning for LLM Agents Policy Optimization Language Model Self-Improvement Policy Learning Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Large language model agents increasingly rely on compound programs for retrieval, tool use, reasoning, and verification, yet their failures often arise from local procedural decisions. Existing reinforcement-learning and prompt-optimization approaches typically rely on scalar rewards or repeatedly modify entire prompts, making it difficult to capture and reuse procedural improvements while preserving a frozen agent. To address this problem, We propose Natural-Language Policy Gradients (NLPG), an external policy-memory method for improving a fixed agent without changing its model parameters or program structure. NLPG diagnoses execution traces, propagates downstream feedback backward through the module graph, and converts recurring failures into route-local natural-language corrections that are aggregated into bounded policy updates for subsequent executions. Across six benchmarks covering memory, reasoning, instruction following, and evidence verification, NLPG also outperforms the strongest listed baseline for each benchmark by 8.71 percentage points on average. These results provide evidence that evaluated procedural experience can be transformed into local and interpretable policy updates, enabling continual improvement of frozen agents.

Abstract

RL for LLM Agents

Explore similar work

NLPG: Natural-Language Policy Gradients for Self-Evolving Language Agents

LangMARL: Natural Language Multi-Agent Reinforcement Learning

Learning Agentic Policy from Action Guidance