RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models

Andrew Zhuoer Feng; Cunxiang Wang; Bosi Wen; Yidong Wang; Yu Luo; Hongning Wang; Minlie Huang

cs.CLFeb 28, 2026

RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models

Authors: Andrew Zhuoer Feng, Cunxiang Wang, Bosi Wen, Yidong Wang, Yu Luo, Hongning Wang, Minlie Huang

Organizations: Tsinghua Univeristy · Peking University

Abstract

Large language model alignment via reinforcement learning depends critically on reward function quality. However, static, domain-specific reward models are often costly to train and exhibit poor generalization in out-of-distribution scenarios encountered during RL iterations. We present RLAR (Reinforcement Learning from Agent Rewards), an agent-driven framework that dynamically assigns tailored reward functions to individual queries. Specifically, RLAR transforms reward acquisition into a dynamic tool synthesis and invocation task. It leverages LLM agents to autonomously retrieve optimal reward models from the Internet and synthesize programmatic verifiers through code generation. This allows the reward system to self-evolve with the shifting data distributions during training. Experimental results demonstrate that RLAR yields consistent performance gains ranging from 10 to 60 across mathematics, coding, translation, and dialogue tasks. On RewardBench-V2, RLAR significantly outperforms static baselines and approaches the performance upper bound, demonstrating superior generalization through dynamic reward orchestration. The data and code are available on this link: https://github.com/ZhuoerFeng/RLAR.

Figures & tables

Figure 1: A conceptual comparison between traditional reward modeling and our proposed RLAR framework.

Domain	Dataset	Train Size	Test Size	OOD	Evaluation Metric
Math	GSM8k	1,500	1,319	✗	Numeric Match
	Hendrycks-Math	1,500	500	✗
	AIME-24/25	0	60	✓
Code	LeetCode-Dataset	2,585	223	✗	Unit Test Execution
Code	MBPP	0	974	✓	Unit Test Execution
Translation	Flores-200	3,000	300	✗	Hybrid (LLM-Judge + BLEU)

Table 1: Summary of domains and datasets considered across Math , Code , Translation , and General Conversation.

Figure 2: The Reinforce Learning with Agentic Reward (RLAR) Framework.

Domain	Avg.	Math				Code		Translation				General
Dataset	Score	GSM8k	HEND	AIME24	AIME25	LCode	MBPP	Flore		WMT24		UChat	Avg Len
Metric	-	NEM	Pass@k	NEM	NEM	Pass@k	Pass@k	BLEU-2	LLM-J	BLEU-2	LLM-J	LLM-J	#
Llama-3.1-8B
Base	37.81	60.50	49.80	6.67	0.00	11.21	41.17	21.04	77.27	26.39	79.24	68.97	152.84
SkyLlama	23.47	11.30 $\downarrow$ 49.20	47.60 $\downarrow$ 2.20	3.33 $\downarrow$ 3.34	0.00 -	8.97 $\downarrow$ 2.24	7.80 $\downarrow$ 33.37	21.16 $\uparrow$ 0.12	45.32 $\downarrow$ 31.95	23.23 $\downarrow$ 3.16	54.68 $\downarrow$ 24.56	60.03 $\downarrow$ 8.94	147.39
SeedX	17.29	1.21 $\downarrow$ 59.29	42.20 $\downarrow$ 7.60	6.67 -	0.00 -	0.00 $\downarrow$ 11.21	13.76 $\downarrow$ 27.41	0.00 $\downarrow$ 21.04	23.24 $\downarrow$ 54.03	0.00 $\downarrow$ 26.39	42.76 $\downarrow$ 36.48	58.73 $\downarrow$ 10.24	89.95

Table 2: Main Experiment Results . For each testset, Bold indicates the best result under that metric, and underline indicates the second best within each model group.

\uparrow

Increment and

\downarrow

decrement are calculated relative to the Zero-Shot baseline for Llama and Qwen. LLM-J is short for LLM-judge, and NEM shorts for numeric exact match. HEND shorts for Hendrycks-Math , LCode shorts for LeetCode-Dataset , UChat shorts for Ultra-Chat . The bold benchmarks ( AIME24/25, MBPP, WMT24 ) indicate that they do not include the training data.

Panel of Figure 3 — Figure 3: Ablation studies on Llama-3.1 and Qwen3.

Agentic Selection within RLAR framework
Method	Avg	Precision IF	Math	Safety	Factuality	Focus
SOTA	87.19	57.14	60.00	97.12	85.29	99.13
Random	78.39	40.48	60.00	89.42	75.49	90.43
GPT-4o	88.19	57.14	68.57	96.15	87.25	99.13
GLM-4.6	88.69	57.14	68.57	96.15	88.24	100.0
GPT-5.1	90.44	66.67	71.42	96.15	90.19	100.0

Table 3: Comparison of Agentic Selection vs. Reward Merging on RewardBench-v2 .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Training & Data
Category / Parameter	Value
Total Epochs	5
Global Train Batch Size	256
Max Prompt Length	10,000
Max Response Length	5,000
Truncation Strategy	’error’

Appendix

Table 4: Hyperparameter settings for GRPO training with Qwen3-8B.

Training & Data
Category / Parameter	Value
Total Epochs	5
Global Train Batch Size	256
Max Prompt Length	10,000
Max Response Length	5,000
Truncation Strategy	’error’

Appendix

Table 5: Hyperparameter settings for GRPO training with Llama-3.1-8B.

Type	Metric	Type	Metric
rule_based	Forbidden_Words	rule_based	Stepwise_Completeness
rule_based	Prompt_Adherence	rule_based	Length
rule_based	Numeric_Accuracy	rule_based	Exact_Template_Match
rule_based	Novelty_Penalty	rule_based	Contradiction_Detection
rule_based	Disallowed_Phrase_Penalty	rule_based	Exact_Output_Match
rule_based	No_Unsupported_Claims	rule_based	Reference_Match

Appendix

Table 6: A list of the generated reward function tool names by our code-agent.

Source	Repo Name
( Liu et al., 2024 )	Skywork/Skywork-Reward-V2-Llama-3.1-8B
( Liu et al., 2024 )	Skywork/Skywork-Reward-V2-Qwen3-8B
( Liu et al., 2024 )	Skywork/Skywork-Reward-V2-Llama-3.2-3B
( Liu et al., 2024 )	Skywork/Skywork-Reward-V2-Qwen3-4B
( Cheng et al., 2025 )	ByteDance-Seed/Seed-X-RM-7B
	OpenAssistant/reward-model-deberta-v3-base

Appendix

Table 7: Successfully Top@9 deployed LLM-based reward models.

Task Type	Unmatched(%)
Infilling	$47.4$
Essay Generation	$43.8$
Multi-Turn	$8.8$

Appendix

Table 8: Web Retrieval Page Rank

Category	Avg Pos
Summ	$7.17$
Translation	$2.36$
RLHF	$5.03$
Multi-Turn	$7.61$
Infill/Gen	$3.75$
Math	$6.87$

Appendix

Table 9: Position Ranks

Category	Count
Code-agent scripts (total)	118
Executable	112 (94.9%)
Rule-based	102 (86.4%)
Metric-based	12 (10.2%)
Learned-model–based	4 (3.4%)
Web-agent repos (retrieved)	21

Appendix

Table 10: Summary of reward tool generation outcomes.

Figure 4: Matching tools with source training dataset distribution.

Figure 7: An illustration on the sensitivity to extreme values of reward functions.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 410 papers

OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning
Oct 28, 2025Ziyou Hu, Zhengliang Shi, Minghang Zhu +5Reward Modeling RL for Language Models
Leiden University, Leiden, Netherland · Shandong University, Qingdao, China · Tsinghua University, Beijing, China
UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms
Sep 5, 2026Peng Lai, Yichao Du, Junchao Wu +6Reward Modeling Reward Model Evaluation
Work done during an internship at Alibaba Cloud. · Southern University of Science and Technology · Wuhan University +4
Dependency-Aware Reward Shaping for Agentic Reinforcement Learning
Oct 1, 2026Ziyi Chen, Yan Zhang, Jianhui Wei +2RL for Language Models Agentic RL
University of Illinois Urbana-Champaign · National University of Singapore · Zhejiang University +1

Oct 28, 2025J/K move · Enter open · S save

OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning

Ziyou Hu, Zhengliang Shi, Minghang Zhu, Haitao Li+4

Leiden University, Leiden, Netherland · Shandong University, Qingdao, China · Tsinghua University, Beijing, China

Reward Modeling RL for Language Models LLM Alignment

Open paper AskPDF arXiv

Reward models (RMs) have become essential for aligning large language models (LLMs), serving as scalable proxies for human evaluation in both training and inference. However, existing RMs struggle on knowledge-intensive and long-form tasks, where evaluating correctness requires grounding beyond the model's internal knowledge. This limitation hinders them from reliably discriminating subtle quality differences, especially when external evidence is necessary. To address this, we introduce OpenRM, a tool-augmented long-form reward model that systematically judges open-ended responses by invoking external tools to gather relevant evidence. We train OpenRM with Group Relative Policy Optimization (GRPO) on over 27K synthesized pairwise examples generated through a controllable data synthesis framework. The training objective jointly supervises intermediate tool usage and final outcome accuracy, incentivizing our reward model to learn effective evidence-based judgment strategies. Extensive experiments on three newly-collected datasets and two widely-used benchmarks demonstrate that OpenRM substantially outperforms existing reward modeling approaches. As a further step, we integrate OpenRM into both inference-time response selection and training-time data selection. This yields consistent gains in downstream LLM alignment tasks, highlighting the potential of tool-augmented reward models for scaling reliable long-form evaluation.

Abstract

RL for Language Models

Explore similar work

OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning

UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms

Dependency-Aware Reward Shaping for Agentic Reinforcement Learning