cs.LGOct 3, 2026
SaveA Bird's-Eye View of Iterative Reward Design
Organizations: Anthropic Fellows Program · University of Edinburgh · Anthropic
Abstract
Designing effective reward functions in RL typically requires substantial expertise and trial and error. Recent work automates this process with LLM-based systems that generate and iteratively improve reward code using policy feedback. However, these methods are often hard to compare because they differ in implementation details, feedback assumptions, and evaluation environments. To address this, we introduce a Benchmark for Iterative Reward Design (BIRD) that expresses existing methods in a unified configuration and evaluation space. This lets us compare algorithms directly, ablate individual design choices, and prototype new components under matched feedback conditions and policy-training budgets. Across MuJoCo, Meta-World, Assistax, and HumanoidBench, we identify a small set of simple design choices that consistently improve performance. Combining these choices yields significantly better performance than the evaluated methods from prior work. Our results highlight the strength of simple baselines and motivate further study of when additional algorithmic complexity improves iterative reward design. Code is available at https://github.com/safety-research/bird.
Figures & tables
Figure 1: Most iterative reward design (IRD) methods follow this basic loop. BIRD uses this shared structure to represent methods in a common format, making them easier to compare.
Figure 2: One iteration of Eureka on Meta-World’s button-press task. The LLM generates eight reward functions, all of which pass verification. Each reward is used to train a PPO policy for 1M environment steps. Reward produces the best policy (89% task success), while produces the worst (0%). Eureka greedily retains and provides the LLM with feedback on how its reward components (e.g., reach , press , and center ) evolve during training. The LLM uses this feedback to revise into . All results are from an actual run.
Figure 3: BIRD represents IRD methods as configurations. Left: Each row is a method and each column is a design choice key, grouped by the stages from Section 2 . Gray cells represent design choices set by the BIRD default (e.g., init=from_scratch ); colored cells are design choices set by the algorithm (e.g., Eureka’s n_candidates=8 ). As we explain in Section 4.3 , ERA-U and ERA-S use the RDA configuration as a backbone. Right: Each configuration is a short YAML file. An excerpt from Eureka’s BIRD configuration.
Figure 4: Comparing IRD methods across four environment suites. Each seed represents one run on an environment. Ticks show mean across tasks and seeds. Bars in this plot and all others are 95% BCa bootstrap confidence intervals. Experimental details are in Appendix I .
Figure 5: Most design choices show no clear benefit. Effects of enabling 30 design choices across applicable baselines (Eureka, RDA, CARD) and nine tasks. Hollow diamonds mark choices introduced in BIRD. Only four choices clearly help: (16) peak own-reward checkpoint, (18) summarized feedback, (19) task-success fitness, and (1) four-turn history. See Appendix E for all choices and experimental details.
Figure 7: ERA-S and ERA-U rank first and second overall. Bradley–Terry ratings on the Elo scale (mean , dashed line) for each benchmark and pooled over all 21 tasks. Within each task, every seed of one method is compared with every seed of another; the higher benchmark score wins and ties count as half a win. Intervals resample seeds with the task set held fixed. Hatched bars mark unsupervised methods. CARD is run only on MuJoCo and MT10.
Figure 8: Examples of high success, low naturalness. All rollouts achieve reference reward. Top left: IRD policies on HumanoidBench flail frenetically on walk and crawl. Top right: scripted policies earn comparable reference reward with even stranger behavior: hopping facing backwards, rocking in place. Bottom: The IRD scratch-itch policy scores well on the reference reward, but poorly on Assistax’s human preference reward: at one point applying over 3 kN of force to the patient’s arm, enough to cause severe pain ( ISO, 2016 ) .
| Compl. | Natur. | |
|---|---|---|
| Rater A vs. B | 0.83 | 0.54 |
| Human vs. benchmark | 0.58 | 0.47 |
| MT10 | 0.78 | 0.71 |
| Gym MuJoCo | 0.57 | – |
| Assistax | 0.36 | 0.29 |
| HumanoidBench | 0.26 | – |
Table 2: Humans agree on completion more than on naturalness. Gym MuJoCo clips were rated for completion only, and HumanoidBench naturalness ratings are almost all at the minimum, so that cell is not reported, as no informative correlation can be calculated.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Term | Definition | Example |
|---|---|---|
| Task objective | The intended behavior the policy should achieve. | Pick up a cube and hold it at the target height for two seconds. |
| Task description | The natural-language statement of the task objective given to the generator, judges, and raters. | “Lift the cube and hold it at the green marker.” |
| Reward function | The function providing the learning signal during policy training. A candidate reward is one proposed during search. | , rewarding reaching, lifting, and maintaining the hold. |
| Reference reward | The reward function distributed with the environment. It is withheld from the generator. | The benchmark’s hand-designed dense reward for the cube task. |
| Benchmark score (fitness) | The value a benchmark assigns a trained policy with respect to the task objective; formally, the fitness . Reward design seeks to maximize . | The rate of successful two-second holds: eight successes in ten evaluation episodes give . |
| Feedback | Information used to guide reward search, including benchmark scores, model or human judgments, and training statistics. | A benchmark score of , or “The cube reaches the target height but is immediately dropped. Increase the reward for maintaining the hold.” |
Table 3: Terminology used throughout the paper. Examples illustrate reward design for a cube-manipulation task.
| Evaluator | |||||||
| Method | Ranking signal | Bench. | VLM | Human | LLM | Feedback | Selection |
| Supervised | |||||||
| Eureka | benchmark | ✓ | – | – | – | S C | argmax |
| DrEureka | benchmark | ✓ | – | – | – | S C | argmax |
| ROSKA | benchmark | ✓ | – | – | – | S C | elitist |
| R ∗ | benchmark | ✓ | – | – | ✓ a | S C | argmax |
Table 4: Methods in BIRD. Feedback: S = score, C = per-component reward statistics, T = trajectories, N = text critique. Optimal Reward Search and Language to Rewards are omitted (no policy-training feedback loop). a LLM-written critics refit reward parameters. b An LLM rates reward designs as a search prior. c The original paper’s own automated variant (REvolve: the benchmark score as its hand-designed fitness; Gran Turismo: an LLM in place of human feedback). d Shown to the generator, but never used to rank. e Interactive variant.
| No. | Design choice | Enabled setting |
|---|---|---|
| Generate | ||
| 1 | Four-turn history | Use a history of four turns as generation context. |
| 2 | Behavioral analysis in prompt | Include behavioral analysis in the generation prompt. |
| 3 | Thought + parent + alignment ‡ | Enable design thought, parent conditioning, and alignment together. |
| 4 | Global-best parent | Use the highest-fitness reward found so far as the parent for generation. |
| 5 | Co-evolve subtasks | Update the subtask decomposition alongside the reward. |
Table 5: Design choices evaluated in the ablation study. Descriptions refer to the enabled setting. Numbering matches Figure 5. indicates design choice introduced in BIRD, indicates multiple key flips to facilitate a design choice
Figure 9: Iteration does not move rewards toward the reference reward. EPIC distance from each iteration’s best reward to the reference reward, over transitions from a competent policy. Lines show means over tasks and ten seeds; bands indicate 95% bootstrap confidence intervals over seeds with the task set held fixed. The dashed line is the median distance of random rewards, averaged over tasks.
Figure 10: Arranging a fixed budget of 16 candidates (Eureka). Each panel is one MT10 task; candidates per iteration trade against iterations, so is a single round. Lines show means over seeds (4–15 per point, median 8); hollow dots show individual seeds; bands indicate 95% bootstrap confidence intervals over seeds. The -axis is the best task success among the first 16 candidates.
| schedule | vs. constant | all 10 tasks | unsaturated 3 |
|---|---|---|---|
| Taper (40, 20, 10, 5, 5) | |||
| Ramp (5, 5, 10, 20, 40) | |||
Table 6: Tapered schedules match constant ones (Eureka, 80 candidates). Mean paired difference in best task success, schedule minus constant arm, paired on task and seed, with 95% bootstrap confidence intervals over seeds (task set fixed). The unsaturated tasks are push, pick-place and peg-insert. Each Taper row pools 46 pairs over all ten tasks and 12 over the unsaturated three; Ramp, 43 and 12; the constant rows, 52 and 12.
Figure 11: Per-task results. Benchmark score of the policy each search would return after each IRD iteration, in each task’s own units: Meta-World MT10 (rows 1–2), MuJoCo (row 3), Assistax (row 4), and HumanoidBench (row 5). Lines show means over seeds (10 per task on MT10 and MuJoCo, 5 on Assistax and HumanoidBench) and shaded bands 95% BCa bootstrap intervals. Dashed lines mark unsupervised methods. CARD, shown on MT10 and MuJoCo, trains policies in its first four iterations only.
Figure 12: Performance against environment steps. The per-task curves of Figure 11 , plotted against cumulative environment steps (log scale, mean over seeds). CARD trains one candidate reward per iteration and the other methods train eight, so CARD finishes with about one-tenth of their environment steps.
| Benchmark | Tasks | Horizon | Benchmark score | Reference reward | Seeds |
|---|---|---|---|---|---|
| Meta-World MT10, v3 ( Yu et al., 2019 ) | button-press-topdown, door-open, drawer-close, drawer-open, peg-insert-side, pick-place, push, reach, window-close, window-open | 500 | Fraction of evaluation episodes in which Meta-World’s own success flag is set on at least one step | Meta-World’s shipped dense reward: shaped reach, grasp and place terms, scaled to per step | 10 |
| Gym MuJoCo, v5 ( Todorov et al., 2012 ) | HalfCheetah, Hopper, Swimmer, InvertedPendulum, Reacher | 1000 | A physical quantity per task (forward displacement; time balanced; final distance to target), normalized so that a uniform-random policy scores 0 and the best method’s mean scores 1 | Gymnasium’s shipped reward: forward velocity less a control cost (Hopper adds an alive bonus); per upright step (InvertedPendulum); negative distance to target less a control cost (Reacher) | 10 |
| Assistax ( Hinckeldey et al., 2025 ) | arm manipulation, feeding, scratch itch, teeth brushing, bed bathing † | 1000 | Fraction of steps on which the tool is at its target and a contact-force condition holds (our metric, defined per task in the text) | Assistax’s shipped dense reward, transcribed term by term: distance-to-target shaping with orientation, speed and contact-force terms, and a control cost | 5 |
| HumanoidBench ( Sferrazza et al., 2024 ) | walk, crawl | 1000 | Mean forward speed as a fraction of the 1 m/s target, clipped to ; for crawl, multiplied by the fraction of steps spent inside the tunnel | HumanoidBench’s shipped reward: posture terms (upright and standing for walk; low body height and in-tunnel for crawl) times velocity toward 1 m/s times small control effort; a return of 700 counts as success | 5 |
Table 7: Benchmarks and tasks. The IRD comparison uses 21 tasks: 10 from Meta-World MT10, 5 from Gym MuJoCo, 4 from Assistax and 2 from HumanoidBench. Horizon is the episode length in environment steps; seeds is the number of independent reward searches per method and task. † Bed bathing is not in the IRD comparison, because its episodes can end early; it appears only in the human study ( Section 4.4 ).
| method | trainings | ranking signal | ||
| supervised | ||||
| Eureka ( Ma et al., 2024a ) | 5 | 40 | reference reward b | |
| RF-Agent ( Gao et al., 2025 ) | 8 | 40 | reference reward b | |
| REvolve ( Hazra et al., 2025 ) | 40 | reference reward b | ||
| R ∗ ( Li et al., 2025a ) | 5 | 40 | reference reward b | |
| CARD ( Sun et al., 2025 ) | 1 | 4 | none a | |
Table 8: The compared arms. candidates per iteration, iterations (published run here); trainings are policies trained per search. The ranking signal picks the candidate carried forward; unsupervised arms never see the reference reward or the benchmark score during search. a CARD ranks nothing; its trajectory screen reads Meta-World’s success flag and passes every reward elsewhere. b The return of the reference reward on MuJoCo, Assistax and HumanoidBench; on Meta-World, which ships a success flag, the flag is used instead, and it coincides with the benchmark score there.
| method | environment shown as | prompt memory | repairs | return |
|---|---|---|---|---|
| Eureka | full source | last iteration, 2 turns | 0 | global best |
| RF-Agent | full source | tree path, 4 ancestors | 8 | global best |
| REvolve | Pythonic abstraction | archive | 0 | archive best |
| R ∗ | full source | last iteration | 0 | global best |
| CARD | Pythonic abstraction + template | full dialogue | 10 | final incumbent |
| RDA | full source | rebuilt, 4 turns | 0 | global best |
Table 9: Per-arm search settings, as resolved in the reported runs. Repairs are LLM retries allowed on failed reward candidate code. Return is the reward the search hands to the final retrain.
| # | configuration | change tested | seeds | steps (M) | vs. RDA [95% CI] | ||
|---|---|---|---|---|---|---|---|
| 1 | v1 | Continuous judge score; return the end of the chain; tapered candidate count at 40 trainings | 5 † | 41.3 | |||
| 2 | v2_verify | Demonstration-based verifier before training; early stopping on own reward; 24 trainings | 3 | 10.7 | |||
| 3 | v2_verify_wide | Wide first round, short tail | 3 | 12.1 | |||
| 4 | v2_verify_short | Two rounds only, 16 then 8 candidates | 3 | 11.9 | |||
| 5 | v2_verify_adapt | Verifier sets how many passers to train | 3 | 8.0 | |||
| 6 | v2_verify_kl | KL penalty toward the demonstration policy | 3 | 10.6 | |||
Table 10: The 24 experiments of Figure 6 , by experiment number. is the configuration’s task success (best checkpoint of the returned reward) relative to RDA, averaged over pick-place, push, peg-insert-side and drawer-open. Intervals are 95% BCa over seeds. † Fewer completed seeds on some tasks. marks correspond the filled markers of the figure.
| benchmark | learner and hardware | searches | set total |
|---|---|---|---|
| Meta-World MT10 | PPO (SB3), 8 vCPUs of a 128-vCPU n2d | 800 | 4,200 CPU-cell-h |
| Gym MuJoCo | PPO (SB3), 8 vCPUs of a 128-vCPU n2d | 400 | 1,400 CPU-cell-h |
| Assistax | IPPO, 1 H200 per training | 214 | 1,600 GPU-h |
| HumanoidBench | FastTD3 + SimbaV2, 1 H200 per training | 93 | 2,200 GPU-h |
Table 11: Compute per benchmark. Set totals cover the searches in Figures 4 and 7 ; development runs, pilots and the hill-climb of Section 4.3 are excluded.
| comparison | question | pairs | exact match | mean gap (scale steps) | ||
|---|---|---|---|---|---|---|
| within rater | completion | 19 | 84% | 0.16 | ||
| within rater | naturalness | 16 | 62% | 0.38 | ||
| between raters | completion | 24 | 42% | 0.58 | ||
| between raters | naturalness | 17 | 53% | 0.49 |
Table 12: Rater reliability. Within-rater figures use unannounced repeat showings; between-rater figures use the clips both raters saw. is Krippendorff’s coefficient with an ordinal metric ( Hayes and Krippendorff, 2007 ) , which measures absolute agreement and therefore charges a constant difference in scale use as disagreement; Spearman’s is invariant to it. Intervals are percentile bootstrap, 2000 resamples.
Figure 13: Screenshot of the annotation UI showing an MT10 task
Figure 14: Breakdown of the composition of annotation data.
| domain | clips | tasks | mean benchmark | mean human | 95% CI | |
| Meta-World MT10 | 79 | 10 | 0.67 | 0.69 | ||
| Gym MuJoCo | 41 | 5 | 0.70 | 0.64 | ||
| Assistax | 70 | 5 | 0.24 | 0.45 | ||
| HumanoidBench † | 67 | 2 | 0.33 | 0.10 | ||
| all domains | 257 | 22 | 0.47 | 0.46 | ||
| naturalness, pooled | ||||||
Table 13: Rank agreement between human completion ratings and the benchmark score, by benchmark domain. The benchmark score is the environment’s own measure on the filmed episode, rescaled to ; on HumanoidBench it is the shipped HumanoidBench reward of the filmed episode over the 700-return success bar, clipped. Intervals are 95% percentile bootstrap over 2000 resamples of tasks. † HumanoidBench has only two tasks, too few to resample, so its interval resamples clips and is correspondingly optimistic. The two naturalness rows pool the clips on which the second question was asked; Gym MuJoCo is completion-only by design.
Figure 17: Holding out the development tasks. Pooled Bradley–Terry ratings over all 21 tasks (left), without the four MT10 tasks ERA-U was developed on (drawer-open, peg-insert-side, pick-place, push; middle), and without all seven tasks used at any point during development, which adds MT10 reach, HalfCheetah, and Swimmer (right).
More in this topic
Reward Design for RL
Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.
See all 252 papers