Progress Reward Model for Reinforcement Learning via Large Language Models

Xiuhui Zhang; Ning Gao; Xingyu Jiang; Yihui Chen; Yuheng Pan; Mohan Zhang; Yue Deng

cs.AIOct 15, 2026

Progress Reward Model for Reinforcement Learning via Large Language Models

Authors: Xiuhui Zhang, Ning Gao, Xingyu Jiang, Yihui Chen, Yuheng Pan, Mohan Zhang, Yue Deng

Organizations: Beihang University, 37 Xueyuan Road, Haidian District, Beijing

Abstract

Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks. In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization. Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance. Code is available at https://github.com/deng-ai-lab/PRM4RL.

Figures & tables

Figure 1: Quick view. (a), (b) and (c) shows the structual similarities and differences between LLM-Augmented methods. (d) is the average evaluation results on Metaworld [ 27 ] and Maniskill [ 28 ] .

Figure 2: An overview of our framework. (a) We prompt the LLM using a Chain-of-Thought approach in Python code format. The reasoning process is provided as comments (e.g., using #), while the intermediate steps and final response are represented as executable Python statements. (b) Our framework augments RL in two ways: subtask prior and the PRM rewards.

Figure 3: The learning curves of comparison methods on Metaworld and Maniskill measured by success rate. All experiments are conducted with 5 random seeds.

Figure 4: The evaluation results of comparison methods on Metaworld(With prefix ‘MW’) and Maniskill(with prefix ‘MS’) measured by success rate. All experiments are conducted with 5 random seeds.

Figure 5: The evaluation results of ablation study on Metaworld measured by success rate. All experiment is conducted with 5 random seeds.

Figure 6: Training details of our and our w.o. potential reward.

Figure 7: The evaluation results of generalization experiment. The results are averaged on 5 random seeds and 100 evaluations per seed.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: The evaluation results of generalization experiment.

Hyper-parameter	Value
Discount factor $\gamma$	0.99
Target update frequency	2
Learning rate	$3\times 10^{-4}$
Train frequency	1
Soft update $\tau$	$5\times 10^{-3}$
Gradient steps	1

Appendix

Table 1: Hyper-parameter of SAC algorithm applied to each task.

Hyper-parameter	Value
Discount factor $\gamma$	0.8
# of epochs per update	15
Learning rate	$1\times 10^{-4}$
# of environments	8
Batch size	400
Target KL divergence	0.1

Appendix

Table 2: Hyper-parameter of PPO algorithm applied to each task.

Task	Task Description
button-press	Press a button in y coordination.
door-close	Close a door with a revolving joint by pushing the door’s handle.
drawer-close	Close a drawer by its handle.
faucet-open	Rotate the handle counter-clockwise.
window-open	Push and open a sliding window by its handle.
sweep-into	Sweep a puck from the initial position into a hole.

Appendix

Table 3: Task list of Metaworld

Task	Task Description
LiftCube	Pick up cube A and lift it up by 0.2 meters.
PullCube	Pull a cube to the goal position.
PushCube	Push a cube to the goal position.
PickCube	Pick up cube A and move it to the 3D goal position.
LiftPegUpright	Move a peg laying on the table to any upright position on the table.

Appendix

Table 4: Task list of Maniskill

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 250 papers

RF-Agent: Automated Reward Function Design via Language Agent Tree Search
Feb 27, 2026Ning Gao, Xiuhui Zhang, Xingyu Jiang +3Reward Design for RL RL Control
Beihang University, 37 Xueyuan Road, Haidian District, Beijing
Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimization
Sep 26, 2026Minghao Li, Rui Tan, Ruihang WangReward Design for RL RL Control
Nanyang Technological University · Yunnan University
Generalizable Dense Reward for Long-Horizon Robotic Tasks
Mar 31, 2026Silong Yong, Stephen Sheng, Carl Qi +6Reinforcement Learning Long-Horizon Robotic Manipulation
Carnegie Mellon University · Amazon Robotics · UT Austin

Feb 27, 2026J/K move · Enter open · S save

RF-Agent: Automated Reward Function Design via Language Agent Tree Search

Ning Gao, Xiuhui Zhang, Xingyu Jiang, Mukang You+2

Beihang University, 37 Xueyuan Road, Haidian District, Beijing

Reward Design for RL RL Control LLM-Guided RL

Open paper AskPDF arXiv

Designing efficient reward functions for low-level control tasks is a challenging problem. Recent research aims to reduce reliance on expert experience by using Large Language Models (LLMs) with task information to generate dense reward functions. These methods typically rely on training results as feedback, iteratively generating new reward functions with greedy or evolutionary algorithms. However, they suffer from poor utilization of historical feedback and inefficient search, resulting in limited improvements in complex control tasks. To address this challenge, we propose RF-Agent, a framework that treats LLMs as language agents and frames reward function design as a sequential decision-making process, enhancing optimization through better contextual reasoning. RF-Agent integrates Monte Carlo Tree Search (MCTS) to manage the reward design and optimization process, leveraging the multi-stage contextual reasoning ability of LLMs. This approach better utilizes historical information and improves search efficiency to identify promising reward functions. Outstanding experimental results in 17 diverse low-level control tasks demonstrate the effectiveness of our method. The source code is available at https://github.com/deng-ai-lab/RF-Agent.

Abstract

Reward Design for RL

Explore similar work

RF-Agent: Automated Reward Function Design via Language Agent Tree Search

Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimization

Generalizable Dense Reward for Long-Horizon Robotic Tasks