Temporal Credit Assignment

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 20

All topics

T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
Sep 30, 2026Bo-Wen Zhang, Junwei He, Maoqi Liu +6Agentic RL Reward Shaping
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · ByteDance
Learning Process Rewards via Reasoning State Propagation
Sep 30, 2026Kai Gan, Zi-Hao Zhou, Bo Ye +3Process Reward Models RL for Language Model Reasoning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · Zhongguancun Academy +1
Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learning
Sep 28, 2026Yangyang Ren, Haodong Zhu, Linlin Yang +3Agentic RL Step-Level Credit Assignment in RL
Beihang University · Zhongguancun Academy · Communication University of China +2
A Systematic Benchmark of Explainable Methods for Temporal Attribution in Sequential Recommendation Systems
Sep 23, 2026Akash Pandey, Kanisha Shah, Addrish Roy +5Gradient-Based Attribution Sequential Recommendation
Capital One, AI Foundations San Jose, CA, USA · Capital One, AI Foundations New York, NY, USA · Capital One, AI Foundations McLean, VA, USA +1
Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation
Sep 15, 2026Shiqi Liu, Zeyu He, Letian Tao +9RL for Language Model Reasoning Temporal Credit Assignment
School of Vehicle and Mobility & College of AI, Tsinghua University · 2Didi Voyager Labs, DiDi Autonomous Driving
Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting
Sep 11, 2026Junhao Zhao, David Michael Simberg, Jacob Kang +2Long-Term Time Series Forecasting Backpropagation
University of Maryland, College Park
Learning Length-Extrapolatable Recurrent Models
Sep 8, 2026Hanwen JiangBackpropagation Temporal Credit Assignment
Adobe Research
Phase-and-First-Arrival VLM Feedback for Sparse-Reward Reinforcement Learning in Surgical Manipulation
Sep 7, 2026Wanli Liuchen, Fangyuan Wang, Bin Li +4Reward Shaping Temporal Credit Assignment
The Hong Kong Polytechnic University · The Chinese University of Hong Kong · Mohamed bin Zayed University of Artificial Intelligence +1
RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
Aug 19, 2026Yugu Li, Zehong Cao, Jianglin Qiao +1Agentic RL Reinforcement Learning
School of CSIT · ACFR · School of EECMS
Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning
Aug 13, 2026Yao Zhou, Hang Gao, Fengge Wu +2Group Relative Policy Optimization Step-Level Credit Assignment in RL
Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Models
Aug 7, 2026Renye Yan, Jikang Cheng, You Wu +4Diffusion Model Alignment Temporal Credit Assignment
Peking University Beijing, China · Stanford University Stanford, California, USA
Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusion
Jul 30, 2026Henglin Liu, Fangyuan Kong, Jing Wang +7Diffusion Model Alignment Video Diffusion Models
Tsinghua University · Kling Team, Kuaishou Technology · Sun Yat-sen University
Counterfactual Shapley Credit Assignment
Jul 18, 2026Mingxuan Li, Kai-Zhan Lee, Elias BareinboimCredit Assignment in RL Temporal Credit Assignment
Department of Computer Science, Columbia University, New York, NY, USA
When Dynamics Models Read the Wrong Time Steps: Label-Free Event Credit Re-Anchoring for Robust Global Readouts
Jun 16, 2026Yifan WangLatent Dynamics Modeling OOD Generalization
Department of Mechanical Engineering, McGill University Montréal, QC H3A 2T7, Canada
Trace-Mediated Peak Bias: Bridging Temporal Credit Assignment and Cognitive Heuristics in Deep Reinforcement Learning
Jun 3, 2026Viktor Veselý, Aleksandar Todorov, Erwan Escudie +1Temporal Credit Assignment Temporal-Difference Learning
Department of AI, University of Groningen, The Netherlands
Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition
Jun 1, 2026Pengyang Ling, Jiazi Bu, Yujie Zhou +6Diffusion Model Alignment Group Relative Policy Optimization
University of Science and Technology of China, China · Shanghai Jiao Tong University, China · Fudan University, China +3
Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learning
Apr 22, 2026Aravind Venugopal, Jiayu Chen, Xudong Wu +3Reward Shaping Temporal Credit Assignment
Carnegie Mellon University · The University of Hong Kong · INFIFORCE Intelligent Technology +1
Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO
Feb 6, 2026Yunze Tong, Mushui Liu, Canyu Zhao +7Flow Matching Temporal Credit Assignment
1Zhejiang University, Hangzhou, China · 2Taobao & Tmall Group of Alibaba, Hangzhou, China · 3Tsinghua University, Beijing, China
Teaching signal synchronization in deep neural networks with prospective neurons
Nov 18, 2025Nicolas Zucchet, Qianqian Feng, Axel Laborieux +3Neural Processes Temporal Credit Assignment
Computer Science Department, ETH Zürich · Gatsby Computational Neuroscience Unit, University College London · Friedrich Miescher Institute for Biomedical Research +3
A Gradient-based yet Spike-Timing-Dependent Solution to the Feedback Learning Problem in Neural Microcircuits
Date pendingXiangnan Zhang, Jingxin Liu, Ranqi Lu +6Neural Processes Temporal Credit Assignment
School of Future Technologies, Beijing Institute of Technology, Beijing, 100081, China. · Key Laboratory of Brain Health Intelligent Evaluation and Intervention (Beijing Institute of Technology), Ministry of Education, Beijing, 100081, China. · School of Medical Technology, Beijing Institute of Technology, Beijing, 100081, China. +5

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning

Bo-Wen Zhang, Junwei He, Maoqi Liu, Feiran Li+5

State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · ByteDance

Agentic RL Reward Shaping Step-Level Credit Assignment in RL Temporal Credit Assignment Sparse-Reward RL RL Fine-Tuning+1

Open paper AskPDF arXiv

Reinforcement learning enables large language model (LLM) agents to learn multi-step behaviors through interaction with their environments. However, rewards in many interactive tasks reflect only the final outcome, providing limited guidance on which intermediate decisions advance the task. Successful training trajectories contain intermediate states that can provide supervision for subsequent interactions. We introduce Trajectory-to-Step Policy Optimization (T2SPO), a method that uses past interaction trajectories to provide step-level feedback for policy learning. T2SPO derives remaining-distance targets from successful trajectories and pairs them with representations of the states visited along the way. Conditioned on these examples, a pretrained TabPFN regressor estimates the remaining distance to success at each state of a new rollout. Changes in this distance estimate across consecutive states yield auxiliary credit for agent steps alongside task-level supervision. As training proceeds, newly completed trajectories refresh the estimator's context, incorporating new experience without updating its parameters. Experiments with 1.5B and 7B language models on ALFWorld and WebShop show that T2SPO consistently improves overall task success over GRPO.

Momentum

Related topics

Latest papers 20