Learning from self-sampled data and sparse environmental feedback remains a fundamental challenge in training self-evolving agents. Temporal credit assignment mitigates this issue by transforming sparse feedback into dense supervision signals. However, previous approaches typically depend on learning task-specific value functions for credit assignment, which suffer from poor sample efficiency and limited generalization. In this work, we propose to leverage pretrained knowledge from large language models (LLMs) to transform sparse rewards into dense training signals (i.e., the advantage function) through retrospective in-context learning (RICL). We further propose an online learning framework, RICOL, which iteratively refines the policy based on the credit assignment results from RICL. We empirically demonstrate that RICL can accurately estimate the advantage function with limited samples and effectively identify critical states in the environment for temporal credit assignment. Extended evaluation on four BabyAI scenarios show that RICOL achieves comparable convergent performance with traditional online RL algorithms with significantly higher sample efficiency. Our findings highlight the potential of leveraging LLMs for temporal credit assignment, paving the way for more sample-efficient and generalizable RL paradigms.
Figures & tables
Figure 1: The pipeline of retrospective in-context online learning, where step ② and step ③ represent retrospective in-context learning.
Figure 2: Comparison of error in advantage function estimation. The x-axis represents the number of trajectories used for estimation, while the y-axis shows the mean error between the estimated advantage and the ground-truth advantages. RICL achieves significantly lower error with fewer samples (around 10) compared to Monte Carlo, which requires about 1000 samples for similar accuracy. Additionally, the error of RICL is more stable across trials (lower variance).
Figure 3: Accuracy comparison of ICL and RICL on predicting expert actions in the BabyAI goto scenario across 1000 trajectories. The Base bar shows the zero-shot performance of LLaMA-3.1-8B-Instruct. RICL outperforms ICL by 7.2%, demonstrating the effectiveness of retrospective updates.
Figure 4: Comparison of our method (RICOL) against four baseline algorithms across four BabyAI scenarios. RICOL consistently demonstrates superior sample efficiency, achieving strong performance with significantly fewer interactions. Notably, RICOL outperforms both PPO (10M) and PPO (3B), by over 50× and 10× fewer environment steps, respectively. Compared to Reflexion, an in-context learning method using trajectory-level verbal feedback, RICOL exhibits better convergent performance by leveraging temporal credit assignment (from RICL) and state-specific feedback. Additionally, RICOL surpasses GPT-4o mini, despite using a smaller policy model (LLaMA-3.2-3B-Instruct), underscoring the importance of interactive learning from the environment. As a useful trick to boost performance, we use the real environment rewards as the advantage and apply advantage-weighted regression during the second stage of training, after RICOL completes its predefined training schedule in the first stage.
Figure 5: RICOL employs in-context credit assignment to generate dense learning signals, enabling more sample-efficient policy training. In contrast, RWR lacks credit assignment, depends on strong base policies with high initial success rates, and performs poorly on tasks with sparse rewards.
Figure 6: The performance of RICOL under varying verbal feedback accuracy. The agent is trained with hand-crafted verbal feedback of different accuracy levels. Despite the presence of noise, RICOL maintains strong performance. Note that 50% accuracy corresponds to random feedback.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Involving Env Feedback
Requiring an Extra Reflector
Retrospective Updating
Multi-turn Credit Assignment
SELF-REFINE
✗
✗
✗
✗
STaR, Reflexion
✓
✗
✗
✗
RLMEC, Retroformer
✓
✓
✗
✗
Iterative RPO, RICO-GRPO
✓
✗
✓
✗
RICL (Ours)
✓
✗
✓
✓
Appendix
Table 1: We propose RICL for LLMs to learn from environmental feedback, fundamentally different from intrinsic self-correction work like SELF-REFINE Madaan et al. [2024] . Unlike STaR [ Zelikman et al., 2022 ] and Reflexion [ Shinn et al., 2024 ] , our method utilizes environmental feedback to retrospectively update the policy. Additionally, we do not fine-tune an extra reflector, as done in RLMEC [ Chen et al., 2024 ] and Retroformer [ Yao et al., 2023 ] . By generating dense training signals from sparse environmental feedback, our approach is more sample-efficient than methods that rely solely on sparse preference rewards, such as Iterative RPO [ Pang et al., 2024 ] .
Method
goto
pickup
pick_up_seq_go_to
open
GPU Type
RWR
72
69
232
227
NVIDIA A40 × 2
RICOL
96
95
594
585
NVIDIA A40 × 2
PPO (10M)
258
257
234
198
GeForce RTX 2080 Ti × 1
PPO (3B)
1440
1440
1440
1440
NVIDIA A40 × 4
Appendix
Table 2: Training time (in minutes) required for each algorithm on different tasks, along with the GPU setup used.
Figure 7: An illustration for the 1D Key-Door scenario.
Figure 8: The prompt used for the LLM policy in the 1D Key-Door scenario.
Figure 9: The prompt used for the LLM reflector in the 1D Key-Door scenario.
Figure 10: Screenshots of four BabyAI scenarios, where the agent is partial-observable.
Figure 11: The prompt used for the LLM policy in all the BabyAI scenarios.
Figure 12: The prompt used for the LLM reflector in all the BabyAI scenarios. Only the Feedback part is extracted and used for RICL’s in-context policy updates.
Figure 13: Comparison of critical states identified by RICL and verbal credit assignment. The x-axis represents a sequence of states: the agent first moves toward the key over 9 steps, picks up the key, then moves toward the door over another 9 steps, and finally opens the door. The y-axis indicates the score reflecting how critical each state is, where GT denotes the ground truth. The results show that both algorithms correctly identify the “pick up the key” action as a critical state-action pair. However, only RICL additionally identifies some of the earlier states in the “move to the key” phase as critical.
Number of Sample Trajectories
1,000
10,000
100,000
1,000,000
MSE on A(s,a)
0.1287
0.0492
0.0492
0.0454
MSE on Δ(s0,feedback)
0.2763
0.2788
0.2713
0.2641
Appendix
Table 3: Approximation errors vs. number of sample trajectories.
LLM agents are commonly trained with RLVR, but the sparse nature of typical RLVR rewards makes credit assignment challenging, especially for long-horizon tasks. We propose Self-Credit, an end-to-end agentic RL method that creates dense supervision signals from sparse outcome rewards using self-generated credit assignment via intra-group rescaling. Self-Credit builds on group-based RL with three components. First, during its rollouts, the policy model generates summaries with citations to earlier turns containing critical evidence, which serve as self-generated credit. Second, we refine this model-generated credit with intra-group rescaling, which compares cited actions across successful and failed rollouts for the same prompt. This approach uses group-level agreement and disagreement to remove undeserved credit and further emphasize credit assigned to critical actions. Third, advantage reweighting uses the rescaled credit weights to update the advantages of the cited turns. On BrowseComp-Plus, Self-Credit improves over the base models by 37.0 and 36.2 points on Qwen3 8B and 32B respectively, and outperforms the strongest agentic RL baselines by 8.5 and 5.6 points, respectively. Self-Credit also improves over the strongest baselines by 4.9 points on GAIA and 4.7 points on xBench-DeepSearch. Analyses show that self-generated credit provides useful signals for selecting critical actions, and that intra-group rescaling, especially on failed rollouts, is important across model scales.
Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assigning credit to individual actions is difficult. Methods such as GRPO assign the same trajectory-level advantage to all actions, while critic-based methods require learning an additional value function that can be costly and unstable. We introduce LLM Value Function RL (LVRL), which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic. LVRL conditions the value model on group context and calibrates its predictions using known values at the initial and terminal states, enabling step-level credit assignment from estimated value differences. We further extend this idea to learning from an expert policy through on-policy distillation (OPD), introducing LLM Value Function OPD (LVOPD), which focuses teacher supervision on actions associated with decreases in estimated value. We evaluate our approach across diverse agentic benchmarks: AppWorld, BabyAI, and SWE-bench Verified. Across policy scales (4B to 32B), LVRL consistently and substantially outperforms both GRPO and PPO. On AppWorld and SWE-bench Verified, LVRL yields absolute improvements of up to 13.6 and 8.6 percentage points over GRPO. Notably, on tasks requiring extended exploration like BabyAI, LVRL bridges the gap from total failure to 75.0% success. Furthermore, LVOPD yields consistent improvements over standard distillation baselines on AppWorld. Finally, Monte-Carlo analysis confirms that our frozen critic accurately identifies semantic pitfalls and recoveries, proving that LVRL extracts robust, policy-aware step-level credit from sparse terminal outcomes.
Group-based reinforcement learning enables large language model agents to learn from sparse outcome rewards in long-horizon tasks. Recent methods derive step-level credit by comparing observation-matched experiences across trajectories. However, identical observations can occur in different contexts, and matched peers are not equally informative for a given decision. Moreover, favorable terminal outcomes can obscure redundant actions or detours within successful trajectories. To address these two challenges, we propose Contextual Credit Policy Optimization (CCPO). For the first, CCPO weights observation-matched peers from other trajectories by contextual similarity, computed from frozen LLM representations and accumulated trajectory statistics, allowing peers from similar contexts to contribute more to a step's baseline. For the second, CCPO pairs historical credit, which compares the realized return against the weighted peer baseline, with future credit, which measures the change in contextual outcome value over a short realized continuation. A step is thus credited for the progress it makes, not only for its trajectory's eventual outcome. Both credits are computed in hindsight from rollouts already collected for training, enabling CCPO to address these challenges without an auxiliary value critic. Across two Qwen2.5-Instruct backbones (1.5B and 7B), CCPO achieves up to 97.4% success on ALFWorld and 79.4% on WebShop and outperforms GRPO by up to 21.6 and 18.5 points. Ablations show that contextual weighting outperforms uniform weighting, and that historical and future credit serve as complementary learning signals.
Huiwen Han, Ziyao Zeng, Zihao Zhu +6
Stanford University · Texas A&M University · Yale University +3