Policy Learning

Follow this topic Ask about this topic

Momentum

79 papers in the last four weeks, up 316% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 416

All topics

StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning
May 26, 2026Yanfei Zhang, Xu Lin, Chenglin WuCredit Assignment in RL Step-Level Credit Assignment in RL
Independent Researcher · Tencent · DeepWisdom
Completion vs Optimality: Policy Gradient in Long-Horizon Cumulative-Damage Problems
May 26, 2026Wolfgang Maass, Sabine JanzenPolicy Gradient Proximal Policy Optimization
Saarland University & German Research Center for Artificial Intelligence (DFKI) · German Research Center for Artificial Intelligence (DFKI)
Credit Assignment with Resets in Language Model Reasoning
May 25, 2026Ankur Samanta, Akshayaa Magesh, Ayush Jain +7RL for Language Model Reasoning Credit Assignment in RL
1Meta AI · 2Columbia University · *Work done at Meta +2
Not only where, But when: Temporal Scheduling for RLVR
May 25, 2026Jinghao Zhang, Ruilin Li, Feng Zhao +1RL for Language Model Reasoning Credit Assignment in RL
University of Science and Technology of China · Shanghai Innovation Institute · Wuhan University
Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimation
May 25, 2026Junwei Zhou, Yu-Wing Tai3D Spatial Reasoning 3D Scene Layout Generation
Department of Computer Science Dartmouth College
Active Learning for Stochastic Contextual Linear Bandits
May 24, 2026Emma Brunskill, Ishani Karmarkar, Zhaoqi LiActive Learning Adaptive Sampling
Stanford University
How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis
May 23, 2026Rei Higuchi, Ryotaro Kawata, Akifumi Wachi +3Reward Modeling Representation Learning
The University of Tokyo · RIKEN AIP · LY Corporation
ECHO: Terminal Agents Learn World Models for Free
May 23, 2026Vaishnavi Shrivastava, Piero Kauffmann, Ahmed Awadallah +1World Model Learning Computer-Use Agents
Microsoft Research
Learning Kernel-Based MDPs from Episodic Preferential Feedback
May 22, 2026Nikola Pavlovic, Sattar Vakili, Qing ZhaoPairwise Preference Learning Regret Minimization in RL
Cornell University · MediaTek Research
Goal-Conditioned Agents that Learn Everything All at Once
May 22, 2026Michael Matthews, Matthew Jackson, Michael Beukman +5Policy Learning Sample-Efficient RL
University of Oxford · 2McGill University · 3MIT +1
From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning
May 22, 2026Ranxu zhang, zeyang li, Jiacheng Huang +5Agentic RL Reinforcement Learning
University of Science and Technology of China · Alibaba Group
Vector Policy Optimization: Training for Diversity Improves Test-Time Search
May 21, 2026Ryan Bahlous-Boldi, Isha Puri, Idan Shenfeld +6Multi-Objective Reinforcement Learning Reinforcement Learning
1MIT · 2Improbable AI Lab · 4Sakana AI +1
Abstraction for Offline Goal-Conditioned Reinforcement Learning
May 21, 2026Clarisse Wibault, Alexander Goldie, Antonio Villares +2Hierarchical RL State Abstraction
FLAIR, MLRG · University of Oxford · FLAIR +1
Factored Diffusion Policies:Compositionally Generalized Robot Control with a Single Score Network
May 21, 2026Sayan Mitra, Ege Yuceel, Noah Giles +1Robotic Control Diffusion Policy
University of Illinois Urbana-Champaign
Chebyshev Policies and the Mountain Car Problem: Reinforcement Learning for Low-Dimensional Control Tasks
May 21, 2026Stefan Huber, Hannes Unger, Georg Schäfer +1RL Control Policy Optimization
Josef Ressel Centre for Intelligent and Secure Industrial Automation, University of Applied Sciences, Salzburg, Austria
OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning
May 21, 2026Yu Li, Rui Miao, Tian Lan +1Value Function Estimation Token-Level Credit Assignment
1George Washington University · 2The University of Texas at Dallas
Behavior-Consistent Deep Reinforcement Learning
May 20, 2026Marcel Hussing, Liv G. d'Aliberti, Claas Voelcker +2Reinforcement Learning Policy Optimization
University of Pennsylvania · Princeton University · University of Texas at Austin
Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting
May 20, 2026Hojin Ko, Jeonggyu HuhReinforcement Learning Stochastic Optimal Control
Department of Mathematics, Sungkyunkwan University, Suwon, Republic of Korea
DISC: Decoupling Instruction from State-Conditioned Control via Policy Generation
May 20, 2026Hanxiang Ren, Pei Zhou, Xunzhe Zhou +1Language-Conditioned Robot Manipulation Visuomotor Policy Learning
‡Zhejiang University §The University of Hong Kong ¶TranscEngram
AGPO: Adaptive Group Policy Optimization with Dual Statistical Feedback
May 20, 2026Miaobo Hu, Shuhao Hu, Bokun Wang +5RL for Language Model Reasoning Group Relative Policy Optimization
Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100092, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
Jointly Learning Predicates and Actions Enables Zero-Shot Skill Composition
May 20, 2026Benedict Quartey, Sebastian Castro, Eric Rosen +3Robot Skill Learning Generalization in Robotic Manipulation
Brown University · Robotics & AI Institute
Learning to Hand Off: Provably Convergent Workflow Learning under Interface Constraints
May 18, 2026Jiayu Li, Enpei Zhang, Dawei Zhou +2Sequential MARL Sequential Decision Making
Stern School of Business · New York University · Department of Computer Science +2
General Preference Reinforcement Learning
May 18, 2026Muhammad Umer, Muhammad Ahmed Mohsin, Ahsan Bilal +5Multi-Objective Language Model Alignment RL Fine-Tuning
Stanford University · The University of Oklahoma
Unified Walking, Running, and Recovery for Humanoids via State-Dependent Adversarial Motion Priors
May 18, 2026Yidan Lu, Yichao Zhong, Liu Zhao +2Humanoid Robot Locomotion Robot Failure Recovery
The University of Hong Kong, Hong Kong SAR, China.
Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation
May 18, 2026Mingfei SunPolicy Gradient Natural Gradient Methods
The University of Manchester, United Kingdom.
How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
May 17, 2026Minghao Tian, Yunfei Xie, Chen WeiRL for Language Model Reasoning Group Relative Policy Optimization
Rice University
FLASH: Efficient Visuomotor Policy via Sparse Sampling
May 15, 2026Jiaqi Bai, Jindou Jia, Yuxuan Hu +5Flow Matching Long-Horizon Robotic Manipulation
MARS Lab, Nanyang Technological University, Singapore
QuantFPFlow: Quantum Amplitude Estimation for Fokker--Planck Policy Optimisation in Continuous Reinforcement Learning
May 14, 2026Abraham Itzhak WeinbergReinforcement Learning Quantum-Inspired ML
AI-WEINBERG, AI Experts, Tel Aviv, Israel
Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy
May 14, 2026Langzhou He, Junyou Zhu, Yue Zhou +7Agentic RL Token-Level Credit Assignment
University of Illinois Chicago · Potsdam Institute for Climate Impact Research · Technical University of Berlin +3
Collaborative Yet Personalized Policy Training: Single-Timescale Federated Actor-Critic
May 14, 2026Leo Muxing Wang, Pengkun Yang, Lili SuFederated RL Actor-Critic Methods
Northeastern University · Tsinghua University

May 26, 2026, cs.AIJ/K move · Enter open · S save

StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning

Yanfei Zhang, Xu Lin, Chenglin Wu

Independent Researcher · Tencent · DeepWisdom

Credit Assignment in RL Step-Level Credit Assignment in RL Step-Level Credit Assignment On-Policy Distillation RL Post-Training Policy Learning+1

Open paper AskPDF arXiv

Reinforcement learning for multi-turn agents suffers from a credit-assignment mismatch: rewards are sparse and trajectory-level, while success often hinges on a few local decisions. Existing online policy distillation (OPD) provides denser token-level supervision, but typically treats heterogeneous agent trajectories as monolithic strings rather than causal interaction units. We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution. StepOPSD decomposes trajectories into action-centered step segments, rescoring them under hindsight-enriched teacher contexts and converting token-level log-probability gaps into sign-preserving advantage shaping with a normalized per-step credit budget before the GRPO update. Across ALFWorld and Search-QA with Qwen3-1.7B and Qwen2.5-3B-Instruct, StepOPSD attains best or second-best results on subsets most sensitive to local causal errors, including first-place performance on ALFWorld Heat (79.1%), PickTwo (95.0%), Search-QA TriviaQA (61.6%), and tied-best performance on HotpotQA (40.4%). The results further reveal a consistent two-knob law: smaller α_clip acts as a broadly stabilizing local trust region, whereas the optimal global mixing strength λ_mix remains task-dependent. These findings suggest that step-aware distillation is most useful when trajectory-level rewards are weakly aligned with the local action that determines downstream success.

Momentum

Related topics

Latest papers 416