Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models

Wen-Tse Chen; Jiayu Chen; Fahim Tajwar; Hao Zhu; Xintong Duan; Ruslan Salakhutdinov; Jeff Schneider

cs.LGFeb 19, 2026

Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models

Authors: Wen-Tse Chen, Jiayu Chen, Fahim Tajwar, Hao Zhu, Xintong Duan, Ruslan Salakhutdinov, Jeff Schneider

Organizations: Carnegie Mellon University · The University of Hong Kong · Stanford University

Abstract

Learning from self-sampled data and sparse environmental feedback remains a fundamental challenge in training self-evolving agents. Temporal credit assignment mitigates this issue by transforming sparse feedback into dense supervision signals. However, previous approaches typically depend on learning task-specific value functions for credit assignment, which suffer from poor sample efficiency and limited generalization. In this work, we propose to leverage pretrained knowledge from large language models (LLMs) to transform sparse rewards into dense training signals (i.e., the advantage function) through retrospective in-context learning (RICL). We further propose an online learning framework, RICOL, which iteratively refines the policy based on the credit assignment results from RICL. We empirically demonstrate that RICL can accurately estimate the advantage function with limited samples and effectively identify critical states in the environment for temporal credit assignment. Extended evaluation on four BabyAI scenarios show that RICOL achieves comparable convergent performance with traditional online RL algorithms with significantly higher sample efficiency. Our findings highlight the potential of leveraging LLMs for temporal credit assignment, paving the way for more sample-efficient and generalizable RL paradigms.

Figures & tables

Figure 1: The pipeline of retrospective in-context online learning, where step ② and step ③ represent retrospective in-context learning.

Figure 2: Comparison of error in advantage function estimation. The x-axis represents the number of trajectories used for estimation, while the y-axis shows the mean error between the estimated advantage and the ground-truth advantages. RICL achieves significantly lower error with fewer samples (around 10) compared to Monte Carlo, which requires about 1000 samples for similar accuracy. Additionally, the error of RICL is more stable across trials (lower variance).

Figure 3: Accuracy comparison of ICL and RICL on predicting expert actions in the BabyAI goto scenario across 1000 trajectories. The Base bar shows the zero-shot performance of LLaMA-3.1-8B-Instruct. RICL outperforms ICL by 7.2%, demonstrating the effectiveness of retrospective updates.

Figure 4: Comparison of our method (RICOL) against four baseline algorithms across four BabyAI scenarios. RICOL consistently demonstrates superior sample efficiency, achieving strong performance with significantly fewer interactions. Notably, RICOL outperforms both PPO (10M) and PPO (3B), by over

50\times

and

10\times

fewer environment steps, respectively. Compared to Reflexion, an in-context learning method using trajectory-level verbal feedback, RICOL exhibits better convergent performance by leveraging temporal credit assignment (from RICL) and state-specific feedback. Additionally, RICOL surpasses GPT-4o mini, despite using a smaller policy model (LLaMA-3.2-3B-Instruct), underscoring the importance of interactive learning from the environment. As a useful trick to boost performance, we use the real environment rewards as the advantage and apply advantage-weighted regression during the second stage of training, after RICOL completes its predefined training schedule in the first stage.

Figure 5: RICOL employs in-context credit assignment to generate dense learning signals, enabling more sample-efficient policy training. In contrast, RWR lacks credit assignment, depends on strong base policies with high initial success rates, and performs poorly on tasks with sparse rewards.

Figure 6: The performance of RICOL under varying verbal feedback accuracy. The agent is trained with hand-crafted verbal feedback of different accuracy levels. Despite the presence of noise, RICOL maintains strong performance. Note that 50% accuracy corresponds to random feedback.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	Involving Env Feedback	Requiring an Extra Reflector	Retrospective Updating	Multi-turn Credit Assignment
SELF-REFINE	✗	✗	✗	✗
STaR, Reflexion	✓	✗	✗	✗
RLMEC, Retroformer	✓	✓	✗	✗
Iterative RPO, RICO-GRPO	✓	✗	✓	✗
RICL (Ours)	✓	✗	✓	✓

Appendix

Table 1: We propose RICL for LLMs to learn from environmental feedback, fundamentally different from intrinsic self-correction work like SELF-REFINE Madaan et al. [2024] . Unlike STaR [ Zelikman et al., 2022 ] and Reflexion [ Shinn et al., 2024 ] , our method utilizes environmental feedback to retrospectively update the policy. Additionally, we do not fine-tune an extra reflector, as done in RLMEC [ Chen et al., 2024 ] and Retroformer [ Yao et al., 2023 ] . By generating dense training signals from sparse environmental feedback, our approach is more sample-efficient than methods that rely solely on sparse preference rewards, such as Iterative RPO [ Pang et al., 2024 ] .

Method	goto	pickup	pick_up_seq_go_to	open	GPU Type
RWR	72	69	232	227	NVIDIA A40 × 2
RICOL	96	95	594	585	NVIDIA A40 × 2
PPO (10M)	258	257	234	198	GeForce RTX 2080 Ti × 1
PPO (3B)	1440	1440	1440	1440	NVIDIA A40 × 4

Appendix

Table 2: Training time (in minutes) required for each algorithm on different tasks, along with the GPU setup used.

Figure 7: An illustration for the 1D Key-Door scenario.

Figure 8: The prompt used for the LLM policy in the 1D Key-Door scenario.

Figure 9: The prompt used for the LLM reflector in the 1D Key-Door scenario.

Panel of Figure 10 — Figure 10: Screenshots of four BabyAI scenarios, where the agent is partial-observable.

Figure 11: The prompt used for the LLM policy in all the BabyAI scenarios.

Figure 12: The prompt used for the LLM reflector in all the BabyAI scenarios. Only the Feedback part is extracted and used for RICL’s in-context policy updates.

Figure 13: Comparison of critical states identified by RICL and verbal credit assignment. The x-axis represents a sequence of states: the agent first moves toward the key over 9 steps, picks up the key, then moves toward the door over another 9 steps, and finally opens the door. The y-axis indicates the score reflecting how critical each state is, where GT denotes the ground truth. The results show that both algorithms correctly identify the “pick up the key” action as a critical state-action pair. However, only RICL additionally identifies some of the earlier states in the “move to the key” phase as critical.

Number of Sample Trajectories	1,000	10,000	100,000	1,000,000
MSE on $A(s,a)$	0.1287	0.0492	0.0492	0.0454
MSE on $\Delta(s_{0},\text{feedback})$	0.2763	0.2788	0.2713	0.2641

Appendix

Table 3: Approximation errors vs. number of sample trajectories.

Step-Level Credit Assignment in RL

Step-level credit assignment in reinforcement learning links delayed rewards to the actions or reasoning steps that produced them.

See all 165 papers

Self-Credit: Credit Assignment for Long-Horizon Agentic RL via Intra-group Rescaling
Oct 20, 2026Fangcong Yin, Ming Yin, Tanvi Bhandarkar +7Agentic RL Step-Level Credit Assignment in RL
Meta · New York University · Duke University
LLM Value Functions for Credit Assignment in Agentic RL
Oct 11, 2026Kaijie Zhu, Weixiang Yan, Omar Attia +3RL for LLM Agents Agentic RL
Apple
Contextual Credit Policy Optimization
Oct 13, 2026Huiwen Han, Ziyao Zeng, Zihao Zhu +6RL for LLM Agents Step-Level Credit Assignment in RL
Stanford University · Texas A&M University · Yale University +3

Oct 20, 2026J/K move · Enter open · S save

Self-Credit: Credit Assignment for Long-Horizon Agentic RL via Intra-group Rescaling

Fangcong Yin, Ming Yin, Tanvi Bhandarkar, Aashu Singh+6

Meta · New York University · Duke University

Agentic RL Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

LLM agents are commonly trained with RLVR, but the sparse nature of typical RLVR rewards makes credit assignment challenging, especially for long-horizon tasks. We propose Self-Credit, an end-to-end agentic RL method that creates dense supervision signals from sparse outcome rewards using self-generated credit assignment via intra-group rescaling. Self-Credit builds on group-based RL with three components. First, during its rollouts, the policy model generates summaries with citations to earlier turns containing critical evidence, which serve as self-generated credit. Second, we refine this model-generated credit with intra-group rescaling, which compares cited actions across successful and failed rollouts for the same prompt. This approach uses group-level agreement and disagreement to remove undeserved credit and further emphasize credit assigned to critical actions. Third, advantage reweighting uses the rescaled credit weights to update the advantages of the cited turns. On BrowseComp-Plus, Self-Credit improves over the base models by 37.0 and 36.2 points on Qwen3 8B and 32B respectively, and outperforms the strongest agentic RL baselines by 8.5 and 5.6 points, respectively. Self-Credit also improves over the strongest baselines by 4.9 points on GAIA and 4.7 points on xBench-DeepSearch. Analyses show that self-generated credit provides useful signals for selecting critical actions, and that intra-group rescaling, especially on failed rollouts, is important across model scales.

Abstract

Step-Level Credit Assignment in RL

Explore similar work

Self-Credit: Credit Assignment for Long-Horizon Agentic RL via Intra-group Rescaling

LLM Value Functions for Credit Assignment in Agentic RL

Contextual Credit Policy Optimization