Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning

Oriol Corcoll; Qiyao Wei; Ciarán Mark Gilligan-Lee

cs.LGOct 9, 2026

Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning

Authors: Oriol Corcoll, Qiyao Wei, Ciarán Mark Gilligan-Lee

Organizations: Spotify · University of Cambridge

Abstract

Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency. We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome. We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return. To investigate what constitutes a useful explanation, we operationalize three complementary variants inspired by Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations. We evaluate EWM on two controlled long-horizon RL benchmarks, a public sequential-recommendation benchmark, and an industry-scale generative recommender. On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with its largest gains when rewards are most delayed. On KuaiRand, EWM outperforms SFT and LaRe offline and A2C online. Finally, in an offline evaluation of an industry generative recommender, EWM considerably improves Recall and increases behavioral proxies. These results suggest that explanations can provide structured and reusable credit for learning from delayed outcomes.

Figures & tables

Figure 1: Explanatory World Model framework. EWMs take evidence (past events) and a phenomenon (the event to explain), and output a hypothesis/explanation about the events causing the phenomenon.

Figure 2: Traditional credit assignment (red) propagates a reward to temporally adjacent events, regardless of their relevance. EWM’s conservative hierarchical operator (CHO, purple) instead uses the explanatory power of each event, recursively, to attribute the reward.

Variant	Criterion	Operationalization
B	Neutral explanation	Select plausible contributors to the outcome; return fewer or no events when evidence is weak.
HP	Counterfactual relevance	Test whether removing or weakening an event makes the outcome less expected; omit redundant events on the same explanatory path.
D	Hard-to-vary explanation	Require the explanation to depend on the specific history, behavior, and target. Replacing these details should weaken it; category matching alone is insufficient.
LB	Explanatory breadth	Infer stable preferences or patterns supported across the history that explain and generalize beyond the outcome without covering unrelated behavior.
LS	Explanatory simplicity	Prefer few underlying preferences or patterns; select representative events and distinguish foundational evidence from later reinforcement.
LI	Integrated explanation	Combine breadth, simplicity, specificity, and behavioral grounding; explain the outcome, support generalization, and identify what would weaken the explanation.

Table 1: Operationalization of explanations in each prompt variant.

Figure 3: Crafter mean episode return at episode-length caps $L\in{150,300,500,1000}$ . Vanilla PPO stays near the random-policy floor. LaRe recovers some delayed credit; EWM is above LaRe at every horizon, and the gap widens as $L$ grows.

Figure 4: WordCraft returns. The 2-mix variant can easily be solved by every algorithm. The 3-mix variant is considerably more difficult due to longer causal chains and only EWM can solve it.

Figure 5: LB achieves the largest mean AUC improvement and log-loss reduction, whereas LaRe differs most from uniform on the four reported ranking metrics.

Figure 6: The termination rule that 5-window-EWM discovered outperforms the baseline vanilla A2C method, and the full-history EWM comes close to the oracle termination rule results.

Metric	EWM RLFT vs baseline ( $95\%$ CI)
Recall	$3.75\times\ [2.62,5.79]$
Converted	$3.86\times\ [2.11,10.0]$
Broad reach ( $\geq$ 2 streams)	$4.00\times\ [2.18,11.5]$

Table 2: Offline evaluation of EWM-reach RLFT against the SFT baseline. Recall is the rate of reproducing the logged recommendation; converted and broad reach additionally require that action to be followed by a further stream or two. All rows are fold-changes against the baseline.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Three Crafter episodes with the same conserved return. Each row is that episode’s step axis, compressed: cited steps keep a few real neighbours, and each long uncredited run collapses to three dots labelled with the number of hidden steps. Solid orange arrows are depth- $1$ hops from the episode end onto the achievements that earned the reward; dashed purple arrows are depth- $2$ hops from those achievements onto earlier enabling actions.

Figure 8: MovieLens-1M policy learning and an illustrative credit allocation. (a) Mean evaluation return across all seeds; shading shows the seed minimum–maximum range, not confidence intervals. We see that the performance of EWM outperforms baselines dense feedback, uniform redistribution, lump terminal reward, and an ablation where we anonymize all the item names in the EWM prompt (so that the EWM does not get any information about the user). Here, each standard deviation is across 100 evaluation episodes. (b) An example session (user 4707, row 573; 30 recommendations in original order), where training rewards and evaluation scores increase to the left. We can see that EWM predictions happen to line up quite well with MF predictions, especially when MF and DeepFM issues different predictions about the same episode.

Figure 9: Anonymized KuaiRand PPO results for Dense, Lump, Uniform and EWM. Each point is a statistic from the same 100-episode evaluation. Total steps denotes the mean steps per episode, not training steps or the sum across evaluation episodes. Eval return is mean cumulative reward per episode. Similar to the anonymization ablation experiment in the MovieLens setting, we again see here that EWM performs worse than dense baselines in the KuaiRand environment when the environment does not provide information about the recommendation items in the dataset. In other words, the EWM sees limited semantic information, because the EWM prompt primarily represents videos through anonymous item and tag identifiers, which limits the usefulness of the LLM’s semantic prior knowledge. This interpretation is consistent with the MovieLens anonymization results.

RL for Recommendation

See all 15 papers

RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback
Sep 27, 2026Jingyi He, Nier Wu, Shuang Liu +3Reward Modeling Counterfactual Explanations
The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University · Carnegie Mellon University +2
Visual Causal-chain Bookmarking: Retrieval-Weighted Credit Assignment for Reinforcement Learning from Delayed Reward
Oct 16, 2026Prathap SelvakumarCredit Assignment in RL Step-Level Credit Assignment in RL
University of Manchester
Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learning
Apr 22, 2026Aravind Venugopal, Jiayu Chen, Xudong Wu +3Reward Shaping Temporal Credit Assignment
Carnegie Mellon University · The University of Hong Kong · INFIFORCE Intelligent Technology +1

Sep 27, 2026J/K move · Enter open · S save

RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback

Jingyi He, Nier Wu, Shuang Liu, Xin Wang+2

The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University · Carnegie Mellon University+2

Reward Modeling Counterfactual Explanations

Open paper AskPDF arXiv

Reward models (RMs) are a key component of large language model post-training, providing reward signals for subsequent reinforcement learning. However, conventional discriminative RMs typically output only scalar scores, making it difficult to identify the response behaviors associated with their scoring decisions. Existing interpretation methods often rely on predefined high-level attributes and require repeated counterfactual interventions for each response pair to validate candidate explanations, lacking a closed-loop mechanism that uses RMs' feedback to train a reusable explainer. To address this, we propose RewardExplainer, a framework that obtains feedback from the target reward model through counterfactual rewriting and uses this feedback to further optimize the explainer. RewardExplainer generates open-ended, atomic, and intervenable natural-language scoring mechanisms, making explanations more concrete, readable, and actionable. It further converts counterfactual feedback into preference supervision, enabling the explainer to more faithfully capture the target RM's scoring preferences and sensitive behaviors than single-pass generation. Extensive experiments across multiple target RMs and explainer backbones show consistent improvements. Beyond interpretation, we use the generated mechanisms to identify potential bias patterns and construct targeted debiasing data for fine-tuning the reward model, improving robustness on reward-hacking benchmarks.

Abstract

RL for Recommendation

Explore similar work

RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback

Visual Causal-chain Bookmarking: Retrieval-Weighted Credit Assignment for Reinforcement Learning from Delayed Reward

Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learning