Reward Engine: Reward-Centric Self-Evolution for Open-Ended Tool Agents

Litong Feng; Ji Qi; Yiran Cheng; Shaowu Pan; Xinyu Zhang; Qijun Li

cs.AIOct 16, 2026

Reward Engine: Reward-Centric Self-Evolution for Open-Ended Tool Agents

Authors: Litong Feng, Ji Qi, Yiran Cheng, Shaowu Pan, Xinyu Zhang, Qijun Li

Organizations: Huawei openJiuwen Team · Department of Statistics and Data Science, The Chinese University of Hong Kong (CUHK)

Abstract

Open-ended tool agents can complete the same objective through different tool paths, and their quality depends jointly on the interaction trace, environment state, and final artifacts, so such tasks rarely admit an executable ground-truth reward. The central challenge is therefore to construct a reward from existing experience that exposes the current policy's failures while remaining stationary during optimization. We introduce Reward Engine, which, like a self-motivated student, first works out what success requires and then optimizes toward it. From the task and initial trajectories, it builds a RewardPack of testcases for required correctness and rubrics for graded quality, each executed as a check that returns a judgment with evidence. The frozen RewardPack then guides the agent to revise and select its outputs, and can also serve as a reward for reinforcement learning. On WorkBuddy Bench Office and Claw-Eval, the constructed reward largely agrees with the benchmark graders, not reversing their ordering on 82.3% and 80.5% of candidate pairs. Used to revise executions, it raises three-run consistency ( $\mathrm{Pass}^{3}$ ) over Baseline from 45.0% to 54.0% and from 78.4% to 83.7%.

Figures & tables

Figure 1: From a self-motivated student to a self-motivated agent. (a) A student facing a new task identifies hard constraints and room to excel. (b) Testcases (required correctness) and rubrics (graded quality), bound to executable checks, form a frozen RewardPack. The reinforcement-learning branch is a possible extension and is not evaluated.

Figure 2: Reward Engine in three stages. 1. Build reward (blue): criteria and executable checks are tested and repaired before a RewardPack is frozen. 2. Execute by reward (orange): the frozen pack supports revision and selection. 3. Update reward (green): products, receipts, and use-time defects inform the next construction round. Benchmark scores are reserved for evaluation.

Method	Mean	Avg. pass (%)	$\Delta$ mean [95% CI]	Sign $p$
WorkBuddy Bench Office
Baseline	81.05	66.67	—	—
Revision	81.10	70.67	+0.05 [ $-$ 2.42, 2.27]	0.0649
Selected by reward	82.21	71.33	+1.16 [ $-$ 1.60, 3.96]	0.0854
Claw-Eval
Baseline	89.44	89.74	—	—

Table 1: Delivery quality. Brackets show paired 95% confidence intervals for the mean difference from Baseline; Sign

p

is a two-sided sign test on per-task differences. Bold marks the best mean and pass rate. Metric definitions are in Section D.4 .

	WorkBuddy Bench Office		Claw-Eval
Method	$\mathrm{pass@3}$ (%)	$\mathrm{Pass}^{3}$ (%)	$\mathrm{pass@3}$ (%)	$\mathrm{Pass}^{3}$ (%)
Baseline	85.00	45.00	98.08	78.37
Revision	84.00	56.00	98.08	87.50
Selected by reward	86.00	54.00	100.00	83.65

Table 2: Three-run success coverage (

\mathrm{pass@3}

) and consistency (

\mathrm{Pass}^{3}

). Bold marks the best value in each column. Definitions and aggregation are given in Sections D.2 and D.4 .

Figure 3: Benchmark results: (a, b) success consistency, where Always, Sometimes, and Never denote three, one or two, and zero successes within a three-run group; dark segments report

\mathrm{Pass}^{3}

. (c) Relative failure reduction with 95% confidence intervals. (d, e) Sorted per-task score gains of Selected by reward over Baseline, retaining all tasks. Definitions and statistical details are given in Sections D.2 and D.4 .

Quantity	WorkBuddy Bench Office	Claw-Eval
Uniform $\rightarrow$ selected score	80.38 $\rightarrow$ 82.21	89.11 $\rightarrow$ 91.47
Gain over uniform [95% CI]	+1.83 [0.22, 3.61]	+2.36 [1.56, 3.20]
Headroom recovered (%)	25.76	39.40
Selection regret [95% CI]	5.29 [2.92, 8.53]	3.63 [2.81, 4.49]
Best-hit rate (%)	49.33	58.65

Table 3: Selection utility of the frozen reward on the generated candidate pools. Gain is Selected minus uniform choice; headroom is the share of the uniform-to-oracle gap recovered; regret is oracle minus Selected. Policy comparisons are in Table 13 .

WorkBuddy Bench Office
Task set	Agreement (%) [95% CI]	$\tau_{+}$ [95% CI]
All tasks	82.33 [74.67, 89.33]	0.647 [0.493, 0.787]
Distinguishable tasks	84.69 [77.21, 91.50]	0.694 [0.544, 0.830]
Claw-Eval
All tasks	80.53 [74.75, 86.08]	0.611 [0.495, 0.722]
Distinguishable tasks	81.72 [75.96, 87.21]	0.634 [0.519, 0.744]

Table 4: Order agreement and

\tau_{+}

with 95% confidence intervals.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Executable checks a	Evidence receipts b	Counterexample testing c	Regression gate d	Frozen during use e	Reported use
Judges, rubrics, and verifiers
LLM-as-a-judge [ Zheng et al., 2023 ]	No	No	No	No	—	Evaluation
Prometheus [ Kim et al., 2024 ]	No	No	No	No	—	Evaluation
LMUnit [ Saad-Falcon et al., 2025 ]	No	No	No	No	—	Evaluation
Agent-as-a-Judge [ Zhuge et al., 2025 ]	No	Part.	No	No	—	Evaluation
Autorubric [ Rao and Callison-Burch, 2026 ]	No	Part.	No	No	Yes	Evaluation, RL

Appendix

Table 5: Properties of evaluators and reward-construction methods as evaluated in the cited work. Yes: provided; Part.: partially provided (see notes); No: not provided; —: not applicable, because the work only evaluates and nothing is optimized against the evaluator.

Symbol	Meaning
Task and actor
$x$	Public task requirements.
$\mathcal{I}$ , $\Delta\mathcal{I}$	Public information: the task, genuine user updates, permitted public knowledge, the execution history, and $\mathcal{B}$ ; it excludes hidden benchmark answers and grading outputs. $\Delta\mathcal{I}$ is the new public evidence admitted at a round boundary.
$\mathcal{B}$	Initial task-only runs available for construction (three per task).
$\pi$	Actor: a model run in the Actor Harness, held fixed.
$h_{t}$ , $a_{t}$ , $o_{t}$	Actor context, action, and observation at step $t$ .

Appendix

Table 6: Notation for reward construction, execution, and revision.

Record	Producer	Fields
Criterion ( $c_{i}$ )	Jury	Identifier; standard; hard or soft kind with its origin and evidence; applicability and priority rationale; measurement question; subrequirements; fail, partial, and full anchors.
Test binding ( $\kappa_{i}$ )	Builder	Test and criterion identifiers; checker key, which a registry dispatches to a function or model evaluator; arguments; question; rubric; optional tool recipe of tool calls and restricted computations.
Measurement record (receipt $e_{i}$ )	Harness	Artifact and environment snapshots; scope; reward and check versions; input references; procedure type; observations; judgment; execution status.
Scope	Caller	Applicable requirements and conversation, admitted sources, user amendments, and environment assumptions.
Version metadata	QC	Version identity and hash; parent version; changed criteria and checks; fresh measurements; review records bound to the identity, scope, and hash; selection rationale.
Repair ledger ( $Q_{v}$ )	QC	Case identity; material (artifact and history); case type (control, error, or boundary); targeted criterion; reference label; agreement recorded once per version and replayed.

Appendix

Table 7: Records stored in a RewardPack. Every published criterion has at least one bound test; obligations that are deferred or cannot be mapped remain unknown in a separate coverage ledger.

Benchmark	Candidate generation and models	External grading
WorkBuddy Bench Office	Three sequential iterations, each producing task-only, reward-guided revision, and hypothesis-guided candidates; results are aggregated across all three iterations. Actor: DeepSeek V4 Flash.	Official task-specific composite of deterministic rules and a model judge (rule weight 0.70–0.95).
Claw-Eval	Three sequential rounds of fresh candidates with same-task pool, checks, memory, and RewardPack carryover. Actor: DeepSeek V4 Flash; Jury and proposal roles also use MiniMax, Qwen, and GLM models.	Native grader with a custom DeepSeek V4 Flash judge.

Appendix

Table 8: Evaluated configuration. On both benchmarks, the reward-guided revision permits at most two revisions with at most three targeted checks per review, and the task-only and hypothesis-guided candidates have delivery review disabled. Baseline repetition and aggregation are defined in Section D.2 .

WorkBuddy Bench Office
Quantity	Revision	Selected by reward
$\Delta$ mean	+0.05 [ $-$ 2.42, 2.27]	+1.16 [ $-$ 1.60, 3.96]
$\Delta$ avg. pass (pp)	+4.00 [ $-$ 3.33, 11.00]	+4.67 [ $-$ 1.33, 11.00]
$\Delta\mathrm{pass@3}$ (pp)	$-$ 1.00 [ $-$ 10.00, 7.00]	+1.00 [ $-$ 7.00, 8.00]
$\Delta\mathrm{Pass}^{3}$ (pp)	+11.00 [2.00, 21.00]	+9.00 [0.00, 19.00]
Median $\Delta$	+1.12	+0.94

Appendix

Table 9: Paired delivery diagnostics against Baseline. Brackets are 95% intervals from task or scenario resampling. Success-rate differences are percentage points (pp); median differences are score points. Sign and Wilcoxon tests are two-sided and exclude ties. Aggregation and statistical procedures are defined in Sections D.2 and D.4 .

WorkBuddy Bench Office
	Task-only candidate		Revision		Selected by reward
Round	Mean	Pass (%)	Mean	Pass (%)	Mean	Pass (%)
Initial	77.67	64.00	81.67	70.00	81.46	66.00
Final	79.41	62.00	81.30	70.00	83.52	76.00
Claw-Eval
Initial	91.24	94.23	91.13	92.31	92.51	97.12

Appendix

Table 10: Initial round and final round performance on 50 WorkBuddy Bench Office tasks and 104 Claw-Eval tasks. Pass rates use thresholds 80 and 75, respectively.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 289 papers

Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agents
Oct 12, 2026Xingjian Diao, Tianyu Yang, Wenjun Huang +8Credit Assignment Fine-Grained Credit Assignment
Dartmouth College · University of Notre Dame · University of California, Irvine +3
Prompt-Level Reward Specifications for Open-Ended Post-Training
May 28, 2026Zijun Weng, Xiaohui Hu, Shuangyong Song +3Open-Ended Generation Reward Modeling
1Fudan University · 2Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.
EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning
Sep 11, 2026Weiyuan Li, Aili Chen, Xintao Wang +11Reward Hacking Reinforcement Learning
School of Data Science, Fudan University · Shanghai Key Laboratory of Data Science · College of Computer Science and Artificial Intelligence, Fudan University +3

Oct 12, 2026J/K move · Enter open · S save

Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agents

Xingjian Diao, Tianyu Yang, Wenjun Huang, Xiangchi Yuan+7

Dartmouth College · University of Notre Dame · University of California, Irvine+3

Credit Assignment Fine-Grained Credit Assignment RL for Tool Use Step-Level Credit Assignment RL for LLM Agents

Open paper AskPDF arXiv

Reinforcement learning improves tool-use agents, yet fine-grained credit assignment remains challenging even with step-level feedback. An observed error may be inherited from an earlier decision, while multiple responsible decisions may share the same downstream consequence or participate in later recovery. We introduce Error-Chain-Aware Optimization (ECHO), a responsibility-aware credit assignment framework that separates responsibility identification from credit allocation. A frozen judge combines retrospective trajectory analysis with decision-time assessment from the policy-visible prefix to distinguish newly introduced errors, inherited consequences, missed corrections, and successful corrections. ECHO then counts shared downstream consequences once across responsible decisions and constrains rewards for repairing self-induced errors through source-funded recovery, producing bounded action-level supervision alongside terminal supervision. Experiments on six diverse tool-use benchmarks at both 4B and 8B scales show strong and consistent improvements over the compared baselines, with ECHO achieving the highest score on five of six benchmarks at each scale. These results support decision responsibility as a useful basis for fine-grained credit assignment in tool-use agents. Our code is available at https://github.com/xid32/ECHO.

Abstract

Tool-Using Agents

Explore similar work

Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agents

Prompt-Level Reward Specifications for Open-Ended Post-Training

EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning