RF-Agent: Automated Reward Function Design via Language Agent Tree Search

Ning Gao; Xiuhui Zhang; Xingyu Jiang; Mukang You; Mohan Zhang; Yue Deng

cs.AIFeb 27, 2026

RF-Agent: Automated Reward Function Design via Language Agent Tree Search

Authors: Ning Gao, Xiuhui Zhang, Xingyu Jiang, Mukang You, Mohan Zhang, Yue Deng

Organizations: Beihang University, 37 Xueyuan Road, Haidian District, Beijing

Abstract

Designing efficient reward functions for low-level control tasks is a challenging problem. Recent research aims to reduce reliance on expert experience by using Large Language Models (LLMs) with task information to generate dense reward functions. These methods typically rely on training results as feedback, iteratively generating new reward functions with greedy or evolutionary algorithms. However, they suffer from poor utilization of historical feedback and inefficient search, resulting in limited improvements in complex control tasks. To address this challenge, we propose RF-Agent, a framework that treats LLMs as language agents and frames reward function design as a sequential decision-making process, enhancing optimization through better contextual reasoning. RF-Agent integrates Monte Carlo Tree Search (MCTS) to manage the reward design and optimization process, leveraging the multi-stage contextual reasoning ability of LLMs. This approach better utilizes historical information and improves search efficiency to identify promising reward functions. Outstanding experimental results in 17 diverse low-level control tasks demonstrate the effectiveness of our method. The source code is available at https://github.com/deng-ai-lab/RF-Agent.

Figures & tables

Figure 1: Quick view. (a) Reshaping LLM-based Reward Function Design Problem from sequential decision-making view. (b) Abstract pipeline comparison between different approaches. (c) Comparison between different methods on max success rates with the numbers of complete training times.

Figure 2: Illustration of our RF-Agent with (a) an exemplary reward function optimization path from the tree, (b) total tree growth process, and (c) iteration based on MCTS. Please refer to the Appendix B for prompts used in RF-Agent, mainly including initialization, expansion, and other process.

task	locomotion				manipulation			Avg norm score
task	Ant	Anymal	Humanoid	Quadcopter	AllegroHand	FrankaCabinet	ShadowHand	Avg norm score
Sparse	$0.14\scriptstyle{\pm 0.05}$	$-2.05\scriptstyle{\pm 0.24}$	$3.01\scriptstyle{\pm 1.99}$	$-1.35\scriptstyle{\pm 0.04}$	$0.03\scriptstyle{\pm 0.00}$	$0.04\scriptstyle{\pm 0.08}$	$0.04\scriptstyle{\pm 0.00}$	0
Human	$6.75\scriptstyle{\pm 0.30}$	$-0.03\scriptstyle{\pm 0.00}$	$5.89\scriptstyle{\pm 0.81}$	$-0.08\scriptstyle{\pm 0.03}$	$11.41\scriptstyle{\pm 1.62}$	$0.11\scriptstyle{\pm 0.07}$	$8.56\scriptstyle{\pm 1.23}$	1
LLM-based Reward Function Design with GPT-4o-mini
Eureka	$4.87\scriptstyle{\pm 1.34}$	$-0.67\scriptstyle{\pm 0.04}$	$3.29\scriptstyle{\pm 0.19}$	$-0.07\scriptstyle{\pm 0.02}$	$13.87\scriptstyle{\pm 3.04}$	$0.01\scriptstyle{\pm 0.00}$	$10.81\scriptstyle{\pm 1.06}$	0.63
Revolve	$5.16\scriptstyle{\pm 1.37}$	$-0.80\scriptstyle{\pm 0.09}$	$3.12\scriptstyle{\pm 0.33}$	$-0.08\scriptstyle{\pm 0.02}$	$18.40\scriptstyle{\pm 0.97}$	$0.19\scriptstyle{\pm 0.17}$	$10.61\scriptstyle{\pm 1.38}$	0.67

Table 1: Quantitative evaluations on IsaacGym. Bold indicates the best results within the same group. Avg norm score represents the average performance after calculating human normalized scores

(Method-Sparse)/(Human-Sparse)

for each task.

Figure 3: Success rates comparison with standard deviation upper bar on BiDex Expert-Easy/Hard.

Figure 4: Success rates with exploration steps under reward functions by different methods.

Figure 5: Reward function optimization performance with sampling counts.

Figure 6: Ablations on the search method, action types, and reasoning component. We report the percent difference in performance compared to the complete RF-Agent. The experiment is built on Ant, FrankaCabinet, and AllegroHand to cover three task types with 4o-mini as the designer.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Examples of the IsaacGym environment.

IsaacGym Environments

Environment (obs dim, action dim)

Task description

Task evaluation score function

F

Ant (60,8)

To make the ant run forward as fast as possible

cur_dist - prev_dist

Appendix

Table 9

Figure 8: Examples of the Bi-DexHands environment.

Bi-DexHands Environments

Environment (obs dim, action dim)

Task description

Task evaluation score function

F

Over (398, 40)

This class corresponds to the HandOver task. This environment consists of two shadow hands with palms facing up, opposite each other, and an object that needs to be passed. In the beginning, the object will fall randomly in the area of the shadow hand on the right side. Then the hand holds the object and passes the object to the other hand. Note that the base of the hand is fixed. More importantly, the hand which holds the object initially can not directly touch the target, nor can it directly roll the object to the other hand, so the object must be thrown up and stays in the air in the process

1[dist < 0.03]

Appendix

Table 11

Ant	Prefill Tokens	Decode Tokens	Total Cost ($)
Eureka	82458	54505	0.1802868
Revolve	162348	43973	0.202944
Ours-total	257564	69704	0.321828
Ours-action	203282	53462	-
Ours-others	54282	16242	-

Appendix

Table 4: Cost on Ant Environment

Humanoid	Prefill Tokens	Decode Tokens	Total Cost ($)
Eureka	78940	60021	0.191414
Revolve	169545	44118	0.20761
Ours-total	254555	66557	0.31247
Ours-action	204283	50270	-
Ours-others	50272	16287	-

Appendix

Table 5: Cost on Humanoid Environment

Figure 9: Success rates with exploration steps under reward functions by different methods on Bidex-hands.

Figure 10: Reward function optimization performance with sampling counts in Bi-DexHands.

Figure 11: Examples of the best node growth path.

Task	Sparse	Eureka	Revolve	Ours
Lie Down	0.12 $\pm$ 0.03	0.64 $\pm$ 0.08	0.59 $\pm$ 0.11	0.73 $\pm$ 0.08
Patrol and Return	0.05 $\pm$ 0.02	0.52 $\pm$ 0.07	0.63 $\pm$ 0.05	0.82 $\pm$ 0.06

Appendix

Table 6: Comparison of methods on OOD-Ant tasks.

Action Combination	Ant	Allegro Hand
a w/ mutation	5.13(-18%)	18.76(-16%)
b w/ mutation + crossover	5.77(-7%)	21.90(-3%)
c w/ mutation + reasoning	5.94(-4%)	21.26(-5%)
d w/ crossover + reasoning	4.85(-22%)	19.21(-15%)
e w/ mutation + crossover + reasoning	6.08(-2%)	22.19(-2%)
Full Version (w/ all actions)	6.22	22.52

Appendix

Table 7: Detailed ablation study on action combinations.

Task	w/ full thought align	w/o first design thought	w/o second thought align	w/o all
ant	6.22	5.08(-18%)	5.94(-4%)	4.96(-20%)
allegro hand	22.52	18.07(-19%)	18.55(-17%)	14.64(-35%)

Appendix

Table 8: Ablation study on thought alignment components.

Resources	Type	License	URL
Eureka [ 28 ]	Codes for Baseline	MIT	https://github.com/eureka-research/Eureka
Revolve [ 17 ]	Codes for Baseline	Available Online	https://github.com/RishiHazra/Revolve/tree/main
IsaacGym [ 31 ]	Codes for Benchmark	BSD 3-Clause License	https://github.com/isaac-sim/IsaacGymEnvs
Bi-DexHands [ 9 ]	Codes for Benchmark	Apache 2.0	https://github.com/PKU-MARL/DexterousHands
RL-games [ 30 ]	Codes for Algorithm	MIT	https://github.com/Denys88/rl_games

Appendix

Table 9: The licenses and URL of baselines , benchmarks and deployment algorithm.

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 250 papers

Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimization
Sep 26, 2026Minghao Li, Rui Tan, Ruihang WangReward Design for RL RL Control
Nanyang Technological University · Yunnan University
Progress Reward Model for Reinforcement Learning via Large Language Models
Oct 15, 2026Xiuhui Zhang, Ning Gao, Xingyu Jiang +4Reward Design for RL LLM-Guided RL
Beihang University, 37 Xueyuan Road, Haidian District, Beijing
Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning
Apr 15, 2026Shentong MoReward Design for RL LLM-Guided RL
Carnegie Mellon University

Sep 26, 2026J/K move · Enter open · S save

Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimization

Minghao Li, Rui Tan, Ruihang Wang

Nanyang Technological University · Yunnan University

Reward Design for RL RL Control Black-Box Optimization Sample-Efficient RL

Open paper AskPDF arXiv

Designing dense reward functions for low-level reinforcement learning (RL) control remains difficult. Recent work uses large language models (LLMs) to iteratively generate and refine reward functions using policy-training feedback within scripted search algorithms. However, evaluating each candidate requires a full RL training run, making sample efficiency a central challenge for reward search on complex control tasks. To address this limitation, we propose an Agentic Reward Black-box Optimization (ARBO) framework, in which an LLM agent builds the search strategy at run time from an evaluation history maintained as its persistent workspace. The evaluation history comprises two components: observations maintained by the evaluation oracle, including candidate scores, per-term training curves, and error tracebacks; and an agent-maintained belief that records diagnoses and intended next steps. The agent queries both with tools and generates the next batch of reward candidates, rather than generating them in a single pass from a fixed prompt. Across four control domains, ARBO achieves gains of 29.9% in manipulation success rate and 192.8% in power-grid score over baseline means under a shared evaluation budget. Ablations examine each component's contribution and sensitivity to backbone choice.

Abstract

Reward Design for RL

Explore similar work

Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimization

Progress Reward Model for Reinforcement Learning via Large Language Models

Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning