Reward Design for RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 64% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 167

All topics

ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning
Jun 23, 2026Wenyang Hu, Junxiang Jia, Zhen Shu +3RL for Language Models RL for Language Model Reasoning
1National University of Singapore · 2SAP
Learning Process Rewards via Success Visitation Matching for Efficient RL
Jun 22, 2026Raymond Tsao, Andrew Wagenmaker, Sergey LevineProcess Reward Models Credit Assignment in RL
UC Berkeley
Causal Reward World Models: Zero-shot Reward Design for Automated Skill Generation
Jun 22, 2026Yang Yang, Yuchuang Tong, Zhengtao Zhang +6RL for Robotics Robot Skill Learning
Institute of Automation, Chinese Academy of Sciences, Beijing 100049, China · Intelligent Manufacturing Institute, HFUT, Hefei 230051, China · School of Electrical Engineering and Automation, Anhui University, Hefei 230061, China +1
EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
Jun 22, 2026Hongxin Ding, Baixiang Huang, Yue Fang +6RL for Language Models Rubric-Based RL
National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education +2
RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation
Jun 20, 2026Pengzhi Yang, Xinyu Wang, Pengyu Jing +7Long-Horizon Robotic Manipulation Robot Skill Learning
† NUS Human-Centered Robotic Lab · ‡ Booking.com · § University of Cambridge +1
A Reward-Petri-Net Interpretation of Temporal Behavior Trees
Jun 19, 2026Till Schmeil, Günther Waxenegger-Wilfing, Sebastian SchirmerReinforcement Learning Reward Shaping
University of Würzburg, Germany · German Aerospace Center (DLR), Germany
ARCO: Adaptive Rubric with Co-Evolution for Multi-Step LLM-Based Agents
Jun 19, 2026Zihang Tian, Jingsen Zhang, Rui Li +3Process Reward Models Credit Assignment in RL
Renmin University of China
Evolutionary Discovery of Developmental Reward Schedules in Deep Reinforcement Learning
Jun 18, 2026Alan Nadelsticher RuvalcabaReinforcement Learning Sparse-Reward RL
College of Computing at the Georgia Institute of Technology, Georgia, USA · School of Engineering and Applied Science at the University of Pennsylvania, Pennsylvania, USA
ExpRL: Exploratory RL for LLM Mid-Training
Jun 15, 2026Violet Xiang, Amrith Setlur, Chase Blagden +2Reinforcement Learning RL for Language Model Reasoning
1Stanford University · 2Carnegie Mellon University · 3OpenAI +1
Evolutionary Bilevel Reward Shaping for Generalization in Reinforcement Learning
Jun 15, 2026Ekasit Usaratniwart, Xilin Gao, Marc Ong +1Reinforcement Learning Reward Shaping
University of Tsukuba, Tsukuba, Japan · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
ReCal: Reward Calibration for RL-based LLM Routing
Jun 10, 2026Qihang Yu, Hanwen Tong, Zhengqi Zhang +5Multi-Objective Reinforcement Learning RL for Language Models
Zhejiang University · Ant Group · Shanghai AI Laboratory
SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
Jun 9, 2026Qianzhong Chen, Hau Zheng, Justin Yu +8RL for Robotics Reward Modeling
Stanford University · UC Berkeley · Shanghai Jiao Tong University
Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short
Jun 8, 2026Han Zhou, Adam X. Yang, Laurence Aitchison +2Pairwise Preference Learning Reward Modeling
University of Cambridge · 2Mistral AI · *Work done as an AI Scientist Intern at Mistral AI.
ComplexConstraints and Beyond: Expert Rubrics for RLVR
Jun 8, 2026Sushant Mehta, Liudas Panavas, Suhaas Garre +1LLM-as-a-Judge Rubric-Based RL
Surge AI
Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions
Jun 8, 2026Xin Jin, Huanqia Cai, Zhen Li +9Reward Modeling Reasoning Distillation
Z-Image Team, Alibaba Group · VCIP, CS, Nankai University
Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization
Jun 7, 2026Hao Chen, Zhanming Shen, Liyao Li +8Intrinsic Reward Methods Reinforcement Learning
Zhejiang University · The Chinese University of Hong Kong · Eastern Institute of Technology
Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models
Jun 7, 2026Yawen Shao, Jie Xiao, Kai Zhu +6RL for Language Model Reasoning Credit Assignment in RL
University of Science and Technology of China · Tongyi Lab · Northeastern University
RREDCoT: Segment-Level Reward Redistribution for Reasoning Models
Jun 4, 2026Mykyta Ielanskyi, Kajetan Schweighofer, Lukas Aichberger +1Reinforcement Learning RL for Language Model Reasoning
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria · Cognizant AI Lab, San Francisco, USA · Work done while at Johannes Kepler University Linz. +1
Reward hacking in physical reinforcement learning revealed by turbulent drag reduction
Jun 4, 2026Giorgio Maria Cavallazzi, Miguel Pérez-Cuadrado, Alfredo PinelliRL Control Reward Hacking
School of Science and Technology, Department of Engineering, City St. George’s, University of London, London, UK
A Pre-Registered Causal Partition of Self-Consistency Elicitation and Reward Design in RLVR
Jun 4, 2026Yuze GaoReinforcement Learning Self-Consistency Decoding
QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards
Jun 2, 2026Rongzhi Zhang, Rui Feng, Zhihan Zhang +8Reinforcement Learning RL for Language Model Reasoning
Amazon · Georgia Institute of Technology
Mitigating False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learning
Jun 2, 2026Can Lv, Mingju Chen, Heng Chang +1Reward Modeling RL for Language Models
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University · Tsinghua University
Reinforcement Learning from Cross-domain Videos with Video Prediction Model
Jun 2, 2026Zhao Yang, Xinrui Zu, Jacob E. Kooi +5RL for Robotics Imitation Learning
VU Amsterdam, The Netherlands · ISAE-SUPAERO, France
RDA: Reward Design Agent for Reinforcement Learning
Jun 1, 2026Hojoon Lee, Ajay Subramanian, Ben Abbatematteo +4Agentic RL Reinforcement Learning
Meta · Holiday Robotics · Boston Dynamics
Deep Research as Rubric for Reinforcement Learning
May 31, 2026Wangyi Mei, Zhouhong Gu, Zhenhan Bai +9Open-Ended Generation Reinforcement Learning
Fudan University · Xiaohongshu Inc. · Beijing University of Posts and Telecommunications
Safe Equilibrium Policy Optimization for Strategic Agent Policies
May 29, 2026Karthika Arumugam, Kiran Kumar Manku, Amit DhandaMulti-Agent System Optimization Game-Playing Agents
†Amazon, USA
LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning
May 28, 2026Xiaoguang Wu, Zhi Zheng, Hui XiongReward Shaping LLM-Guided RL
University of Science and Technology of China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · 2Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)
Prompt-Level Reward Specifications for Open-Ended Post-Training
May 28, 2026Zijun Weng, Xiaohui Hu, Shuangyong Song +3Open-Ended Generation Reward Modeling
1Fudan University · 2Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.
Label-Free Reinforcement Learning via Cross-Model Entropy
May 27, 2026Matt Gorbett, Hossein ShiraziReinforcement Learning Instruction Following
Independent Researcher · San Diego State University
When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL
May 27, 2026Youting Wang, Yuan Tang, Bowen Liu +2Reward Shaping LLM-Guided RL

Jun 23, 2026, cs.LGJ/K move · Enter open · S save

ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning

Wenyang Hu, Junxiang Jia, Zhen Shu, Daniel Dahlmeier+2

1National University of Singapore · 2SAP

RL for Language Models RL for Language Model Reasoning Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards Policy Learning Reward Design for RL

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) for language-model reasoning can fail at both extremes of task difficulty: easy prompts often produce all-correct, low-diversity rollout groups with little gradient signal, while hard prompts can produce all-incorrect groups with no positive reward. We introduce ExTra (Exploratory Trajectory Optimization), a GRPO-compatible framework that extracts exploration signals from the model's own rollouts. ExTra combines two mechanisms: (i) a novelty reward that adds embedding-based diversity bonuses after GRPO normalization, rewarding diverse correct solutions; and (ii) entropy-guided prefix regeneration, which scores partial trajectories using entropy signals and continues exploration from promising intermediate steps. Across six mathematical reasoning benchmarks, ExTra improves Qwen3-1.7B over GRPO by about +5 points on pass@1 and +7 points on pass@16, showing that trajectory-level exploration signals can improve both single-sample accuracy and inference-time coverage.

Momentum

Related topics

Latest papers 167