Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency. We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome. We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return. To investigate what constitutes a useful explanation, we operationalize three complementary variants inspired by Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations. We evaluate EWM on two controlled long-horizon RL benchmarks, a public sequential-recommendation benchmark, and an industry-scale generative recommender. On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with its largest gains when rewards are most delayed. On KuaiRand, EWM outperforms SFT and LaRe offline and A2C online. Finally, in an offline evaluation of an industry generative recommender, EWM considerably improves Recall and increases behavioral proxies. These results suggest that explanations can provide structured and reusable credit for learning from delayed outcomes.
Figures & tables
Figure 1: Explanatory World Model framework. EWMs take evidence (past events) and a phenomenon (the event to explain), and output a hypothesis/explanation about the events causing the phenomenon.
Figure 2: Traditional credit assignment (red) propagates a reward to temporally adjacent events, regardless of their relevance. EWM’s conservative hierarchical operator (CHO, purple) instead uses the explanatory power of each event, recursively, to attribute the reward.
Variant
Criterion
Operationalization
B
Neutral explanation
Select plausible contributors to the outcome; return fewer or no events when evidence is weak.
HP
Counterfactual relevance
Test whether removing or weakening an event makes the outcome less expected; omit redundant events on the same explanatory path.
D
Hard-to-vary explanation
Require the explanation to depend on the specific history, behavior, and target. Replacing these details should weaken it; category matching alone is insufficient.
LB
Explanatory breadth
Infer stable preferences or patterns supported across the history that explain and generalize beyond the outcome without covering unrelated behavior.
LS
Explanatory simplicity
Prefer few underlying preferences or patterns; select representative events and distinguish foundational evidence from later reinforcement.
LI
Integrated explanation
Combine breadth, simplicity, specificity, and behavioral grounding; explain the outcome, support generalization, and identify what would weaken the explanation.
Table 1: Operationalization of explanations in each prompt variant.
Figure 3: Crafter mean episode return at episode-length caps L∈150,300,500,1000 . Vanilla PPO stays near the random-policy floor. LaRe recovers some delayed credit; EWM is above LaRe at every horizon, and the gap widens as L grows.
Figure 4: WordCraft returns. The 2-mix variant can easily be solved by every algorithm. The 3-mix variant is considerably more difficult due to longer causal chains and only EWM can solve it.
Figure 5: LB achieves the largest mean AUC improvement and log-loss reduction, whereas LaRe differs most from uniform on the four reported ranking metrics.
Figure 6: The termination rule that 5-window-EWM discovered outperforms the baseline vanilla A2C method, and the full-history EWM comes close to the oracle termination rule results.
Metric
EWM RLFT vs baseline ( 95% CI)
Recall
3.75×[2.62,5.79]
Converted
3.86×[2.11,10.0]
Broad reach ( ≥ 2 streams)
4.00×[2.18,11.5]
Table 2: Offline evaluation of EWM-reach RLFT against the SFT baseline. Recall is the rate of reproducing the logged recommendation; converted and broad reach additionally require that action to be followed by a further stream or two. All rows are fold-changes against the baseline.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Three Crafter episodes with the same conserved return. Each row is that episode’s step axis, compressed: cited steps keep a few real neighbours, and each long uncredited run collapses to three dots labelled with the number of hidden steps. Solid orange arrows are depth- 1 hops from the episode end onto the achievements that earned the reward; dashed purple arrows are depth- 2 hops from those achievements onto earlier enabling actions.
Figure 8: MovieLens-1M policy learning and an illustrative credit allocation. (a) Mean evaluation return across all seeds; shading shows the seed minimum–maximum range, not confidence intervals. We see that the performance of EWM outperforms baselines dense feedback, uniform redistribution, lump terminal reward, and an ablation where we anonymize all the item names in the EWM prompt (so that the EWM does not get any information about the user). Here, each standard deviation is across 100 evaluation episodes. (b) An example session (user 4707, row 573; 30 recommendations in original order), where training rewards and evaluation scores increase to the left. We can see that EWM predictions happen to line up quite well with MF predictions, especially when MF and DeepFM issues different predictions about the same episode.
Figure 9: Anonymized KuaiRand PPO results for Dense, Lump, Uniform and EWM. Each point is a statistic from the same 100-episode evaluation. Total steps denotes the mean steps per episode, not training steps or the sum across evaluation episodes. Eval return is mean cumulative reward per episode. Similar to the anonymization ablation experiment in the MovieLens setting, we again see here that EWM performs worse than dense baselines in the KuaiRand environment when the environment does not provide information about the recommendation items in the dataset. In other words, the EWM sees limited semantic information, because the EWM prompt primarily represents videos through anonymous item and tag identifiers, which limits the usefulness of the LLM’s semantic prior knowledge. This interpretation is consistent with the MovieLens anonymization results.
Reward models (RMs) are a key component of large language model post-training, providing reward signals for subsequent reinforcement learning. However, conventional discriminative RMs typically output only scalar scores, making it difficult to identify the response behaviors associated with their scoring decisions. Existing interpretation methods often rely on predefined high-level attributes and require repeated counterfactual interventions for each response pair to validate candidate explanations, lacking a closed-loop mechanism that uses RMs' feedback to train a reusable explainer. To address this, we propose RewardExplainer, a framework that obtains feedback from the target reward model through counterfactual rewriting and uses this feedback to further optimize the explainer. RewardExplainer generates open-ended, atomic, and intervenable natural-language scoring mechanisms, making explanations more concrete, readable, and actionable. It further converts counterfactual feedback into preference supervision, enabling the explainer to more faithfully capture the target RM's scoring preferences and sensitive behaviors than single-pass generation. Extensive experiments across multiple target RMs and explainer backbones show consistent improvements. Beyond interpretation, we use the generated mechanisms to identify potential bias patterns and construct targeted debiasing data for fine-tuning the reward model, improving robustness on reward-hacking benchmarks.
Jingyi He, Nier Wu, Shuang Liu +3
The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University · Carnegie Mellon University +2
RL agents trained on a single trajectory-level return face a credit-assignment problem: when a reward arrives many steps after the decision that caused it, every action gets the same signal regardless of causal responsibility. This paper studies Visual Causal-chain Bookmarking (VCBM), which concentrates credit on the decision points that determine an episode's return, via two instantiations. The first targets a fully observed, low-dimensional MDP (a delayed-reward watch-repair task), where VCBM statistically discovers which state components the agent causally controls and estimates per-decision-point action values directly, evaluated against ground truth and RUDDER, an LSTM-based baseline. VCBM converges to 95.07% correct decisions in a mean of 1,532.3 episodes across 20 seeds, versus RUDDER's slower, more variable convergence (mean 3,327.1 episodes; p<0.001); VCBM's discovered causal structure exactly matches ground truth in all seeds. The second targets a partially observed, language-based setting: an LLM agent trained with Leave-One-Out PPO (LOOP) in AppWorld, where causal discovery is intractable and a structural heuristic (state-mutating tool calls) substitutes for it, with credit distributed via retrieval over episodic memory and blended with LOOP's advantage. Over 200 iterations on identical Qwen2.5-7B-Instruct/LoRA/FSDP2 configurations, VCBM-blended LOOP improves task success over plain LOOP (0.354 vs. 0.458), reduces final-step max importance weight, and improves token efficiency, with a small KL exception favouring LOOP. The watch-repair result supports the claim that weighting causally relevant steps is more sample-efficient than an undifferentiated baseline or a learned return-redistribution network; the AppWorld result, limited to one seed per arm by a 2-GPU budget, is a compute-limited demonstration at a harder, natural-language scale, not a statistical confirmation.
The temporal lag between actions and their long-term consequences makes credit assignment a challenge when learning goal-directed behaviors from data. Generative world models capture the distribution of future states an agent may visit, indicating that they have captured temporal information. How can that temporal information be extracted to perform credit assignment? In this paper, we formalize how the temporal information stored in world models encodes the underlying geometry of the world. Leveraging optimal transport, we extract this geometry from a learned model of the occupancy measure into a reward function that captures goal-reaching information. Our resulting method, Occupancy Reward Shaping, largely mitigates the problem of credit assignment in sparse reward settings. ORS provably does not alter the optimal policy, yet empirically improves performance by 2.2x across 13 diverse long-horizon locomotion and manipulation tasks. Moreover, we demonstrate the effectiveness of ORS in the real world for controlling nuclear fusion on 3 Tokamak control tasks. Code: https://github.com/aravindvenu7/occupancy_reward_shaping; Website: https://aravindvenu7.github.io/website/ors/
Aravind Venugopal, Jiayu Chen, Xudong Wu +3
Carnegie Mellon University · The University of Hong Kong · INFIFORCE Intelligent Technology +1