MLREF: Efficient Module Reuse for Reward Design in Reinforcement Learning via Large Language Models

Chenglin Liu; Xun Wang; Ruishuo Chen; Zhuoran Li; Longbo Huang

cs.LGAug 19, 2026

MLREF: Efficient Module Reuse for Reward Design in Reinforcement Learning via Large Language Models

Authors: Chenglin Liu, Xun Wang, Ruishuo Chen, Zhuoran Li, Longbo Huang

Organizations: Institute for Interdisciplinary Information Sciences, Tsinghua University

Abstract

Reward function design remains a bottleneck in reinforcement learning. While large language models (LLMs) have enabled automated reward generation, existing methods generate and revise reward functions as monolithic programs, making it difficult to reliably preserve and reuse effective components discovered in earlier iterations, leading to unstable performance across iterations. To address this, we propose Module Level Reward Evolution Framework (MLREF). At the core of MLREF is a module pool, a persistent repository of reusable reward components. MLREF treats the module pool as the primary optimization object: the pool evolves across iterations by accumulating successful modules, refining underperforming ones, and reusing proven components; while reward functions are constructed as linear combinations of modules drawn from this pool. To drive this evolution, MLREF integrates three mechanisms: reflection-based refinement, hybrid credit assignment, and a merge strategy with rollback, which together improve the effectiveness and robustness of reward optimization. Experiments on 17 tasks show that MLREF outperforms strong baselines by 25.2% in locomotion and 6.6% in manipulation, with more stable optimization dynamics.

Figures & tables

Property	EUREKA	RF-Agent	MLREF (Ours)
Optimization Unit	Function	Function	Module Pool
Module Identity	–	Anonymous	Named + Spec.
Cross-iter. Persistence	–	Partial	Persistent Pool
Credit Assignment	Holistic	Holistic	Per-Module Hybrid
Failure Recovery	–	Implicit	Explicit Rollback
Semantic Guidance	–	–	Task + Feedback Refl.

Table 1: Comparison of key design properties across LLM-based reward design methods. MLREF is the only framework that treats modules as first-class, persistently managed objects with per-module credit assignment, semantic guidance, and a principled failure recovery mechanism.

Figure 1: Overview of the MLREF pipeline. In the initialization phase (steps 1–3), the LLM performs task and environment reflection and generates diverse module pool variants. In the iterative optimization phase (steps 4–9), each variant assembles a reward function via weighted linear combination for RL evaluation; a merge strategy with rollback consolidates successful modules; and feedback reflection guides the next round of pool improvement.

Locomotion
Task	EUREKA	RF-Agent	MLREF (Ours)
Allegro Hand	25.48 $\pm$ 1.27	27.41 $\pm$ 1.90	28.89 $\pm$ 1.36
Ant	12.35 $\pm$ 1.29	12.34 $\pm$ 0.70	12.47 $\pm$ 0.30
Anymal	-0.0057 $\pm$ 0.0021	-0.0064 $\pm$ 0.0010	-0.0073 $\pm$ 0.0019
Franka Cabinet	0.397 $\pm$ 0.202	0.701 $\pm$ 0.200	0.997 $\pm$ 0.006
Humanoid	7.95 $\pm$ 0.87	5.56 $\pm$ 1.63	7.40 $\pm$ 1.12

Table 2: Performance comparison across 17 tasks. Each row shows the mean and standard deviation of the raw performance metric. The score of locomotion tasks is normalized before averaging. The percentage improvement is relative to the best baseline. Bold indicates the best mean result.

Method	Ant	Block Stack	Bottle Cap	Avg. % of Full
Full MLREF	12.47 $\pm$ 0.30	0.313 $\pm$ 0.244	0.987 $\pm$ 0.012	100.0%
GPT-4o	10.77 $\pm$ 1.95	0.101 $\pm$ 0.042	0.041 $\pm$ 0.027	40.9%
No Pool	11.51 $\pm$ 0.64	0.085 $\pm$ 0.040	0.988 $\pm$ 0.010	73.2%
No Reflection	9.58 $\pm$ 0.55	0.057 $\pm$ 0.032	0.639 $\pm$ 0.401	53.3%
No Weight Opt.	11.68 $\pm$ 1.14	0.044 $\pm$ 0.025	0.829 $\pm$ 0.338	63.9%

Table 3: Ablation results on three representative tasks. Best results are bold.

Panel of Figure 2 — Figure 2: Performance evolution across iterations (best sample per iteration). Left: all-task average. Middle: Shadow Hand. Right: Catch Abreast. Shaded areas denote standard deviation.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

MLREF — Optimization
Parameter	Value
Max iterations $I$	9
Parallel samples per iteration $S$	3
Pipeline runs	3
RL evaluation seeds	5
MLREF — Pool Merge

Appendix

Table 4: Hyperparameters for MLREF and baseline methods.

Panel of Figure 3 — Figure 3: Evolution curves for all 7 locomotion tasks.

Panel of Figure 4 — Figure 4: Evolution curves for all 10 manipulation tasks.

Panel of Figure 5 — Figure 5: Average evolution curves. Left: all 17 tasks. Middle: locomotion (7 tasks). Right: manipulation (10 tasks).

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 252 papers

Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning
Apr 15, 2026Shentong MoReward Design for RL LLM-Guided RL
Carnegie Mellon University
RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models
Feb 28, 2026Andrew Zhuoer Feng, Cunxiang Wang, Bosi Wen +4RL for Language Models Reward Design for RL
Tsinghua Univeristy · Peking University
Progress Reward Model for Reinforcement Learning via Large Language Models
Oct 15, 2026Xiuhui Zhang, Ning Gao, Xingyu Jiang +4Reward Design for RL LLM-Guided RL
Beihang University, 37 Xueyuan Road, Haidian District, Beijing

Apr 15, 2026J/K move · Enter open · S save

Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning

Shentong Mo

Carnegie Mellon University

Reward Design for RL LLM-Guided RL Robotic RL

Open paper AskPDF arXiv

Designing effective reward functions is a cornerstone of reinforcement learning (RL), yet it remains a challenging and labor-intensive process due to the inefficiencies and inconsistencies inherent in traditional methods. Existing methods often rely on extensive manual design and evaluation steps, which are prone to redundancy and overlook local uncertainties at intermediate decision points. To address these challenges, we propose the Chain of Uncertain Rewards (CoUR), a novel framework that integrates large language models (LLMs) to streamline reward function design and evaluation in RL environments. Specifically, our CoUR introduces code uncertainty quantification with a similarity selection mechanism that combines textual and semantic analyses to identify and reuse the most relevant reward function components. By reducing redundant evaluations and leveraging Bayesian optimization on decoupled reward terms, CoUR enables a more efficient and robust search for optimal reward feedback. We comprehensively evaluate CoUR across nine original environments from IsaacGym and all 20 tasks from the Bidexterous Manipulation benchmark. The experimental results demonstrate that CoUR not only achieves better performance but also significantly lowers the cost of reward evaluations.

Abstract

Reward Design for RL

Explore similar work

Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning

RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models

Progress Reward Model for Reinforcement Learning via Large Language Models