WATERFALL: Workflow for Adaptive Training with Evolutionary Reward Formulation and Automated Learning Loops
Organizations: ETH Zurich, Zurich, Switzerland · UCL, London, UK
Abstract
Reward design remains a central bottleneck in reinforcement learning (RL), particularly in sparse-reward, long-horizon and partially observable settings requiring dependency chaining, memory, and deceptive affordance disambiguation. While recent foundation-model approaches reduce manual reward engineering, most assume goal-conditioning, task descriptions, expert demonstrations, or ground-truth metrics. These assumptions are problematic in partially observable environments, where task-relevant objectives, affordances, and dynamics must be discovered through interaction rather than disclosed a priori. We formalise this problem setting by introducing WATERFALL, an iterative population-based workflow for automated reward discovery under the strict observability constraints inherent in POMDP environments. WATERFALL achieves this by (i) synthesising diverse programmatic reward candidates via persona-conditioned generation, (ii) evaluating candidate behaviour from raw visual rollouts utilising a Swiss-system evolutionary tournament judged by vision-language models to identify elites, and (iii) leveraging longitudinal assessment histories to iteratively mutate, escalate, or prune candidates. We evaluate WATERFALL across fully observable and reveal-gated partially observable environments ranging from discrete to continuous, comparing against sparse-reward RL, intrinsic-motivation and foundation-model reward-synthesis baselines. Our results indicate that WATERFALL discovers and refines reward programs without access to privileged information, outperforming baselines that retain their native information advantages, most clearly in the POMDP regime. Extensive ablations substantiate that visual evidence, tournament-based selection, and iterative refinement are jointly important. Enforcing this contract is what separates autonomous reward discovery from benchmark-privileged reward shaping.
Figures & tables
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Specification | Env. I | Env. II | Env. III | Env. IV |
| Environment name | MiniGrid Locked Room | Dark Corridor | Medusa’s Revenge | Reveal-Gated Robotic Button Task |
| Grid size | Continuous physics | |||
| Episode horizon | 320 | 192 | 256 | 10,240 action applications; 1,024 policy decisions |
| Observation Space | ||||
| Observation vector size | 150 | 21 | 25 | 22 |
| Local perception | egocentric rays | T-shaped rays | 8-neighbour ray values | Proprioception and three traffic-light indicators |
| Method Information Privilege Modality Refinement Evolutionary Required Human In-Loop? Goal-Conditioned Ground Truth 11 WATERFALL (ours) No No Vision+Text Iterative Yes (Swiss-Tourn.) No EUREKA [ Ma et al., 2024 ] Yes 1 Yes Text Iterative Yes (Elite/Greedy) 10 No 14 L2R [ Yu et al., 2023 ] Yes 2 Yes 2 Text Iterative No Yes 2 CAR-VLM / VLM-CaR [ Venuto et al., 2024 ] Yes 3 Yes 3 Vision+Text One-shot 12 No No 3 Text2Reward [ Xie et al., 2024 ] Yes 2 Yes Text One-shot/Iter. 4 No No 4 REvolve [ Hazra et al., 2025 ] Yes 1 No 5 Text Iterative Yes (Elo) 6 Yes 5 LaRes [ Li et al., 2025a ] Yes Yes 7 Text Iterative Yes (Evo) No RF-Agent [ Gao et al., 2025 ] Yes Yes 7 Text Iterative No No RE-GoT [ Yao et al., 2025 ] Yes 15 No 8 Vision+Text Iterative No No R* [ Li et al., 2025b ] Yes Yes 7 Text Iterative Yes (Evo) No FORGE (withdrawn) 9 [ Fan and Du, 2025 ] Yes Yes Text Iterative Yes (Evo) No GRACE [ Sapora et al., 2026 ] Yes (Demos) 13 No 13 Text Iterative Yes (Evo) Yes |
| 1 Goal specified via task/env context (text/code), not goal images. 2 “Interactive feedback/optim” shorthand (language goal with an interactive feedback or optimisation loop). 3 Uses goal image and/or trajectory-based verification; no interactive human feedback loop is queried during training. 4 Table entry refers to configurations where iterative refinement can use human feedback (optionally). 5 Fitness depends on (minimal) human preference feedback. 6 Elo-style ranking from pairwise comparisons. 7 Search/controller consumes scalar evaluation/training metrics (e.g., success/return). 8 Uses VLM rollout evaluation rather than scalar GT metrics. 9 Withdrawn ICLR 2026 submission (OpenReview). 10 “Elite/Greedy” denotes elitist selection (keep best-so-far reward/program each iteration). 11 “Ground Truth” denotes exposing ground truth rewards either by explicitly providing reward plots of those rewards or success rates and/or metrics stemming from ground truths. This is particularly the case for methods explicitly relying on collecting such ground truth metrics when performing iterative reward synthesis. 12 One-shot inclusive of a verifier loop. 13 Requires privileged supervision in the form of expert trajectories from an expert policy or human demo. Those demos encode the task objective implicitly (goal/intent). 14 Not inherently required and is optional for RLHF. 15 Explicitly requires environment codes and a text description of the task goal to map out its initial graph-based reasoning paths. Core Baselines Included: RL (sparse ), RL wCBE [ Tang et al., 2017 ] , RL wICM [ Pathak et al., 2017 ] and RL wRND [ Burda et al., 2019 ] , while not part of the aforementioned architectures leveraging reward synthesis, these algorithms were included as part of the core experimental exploration-based baselines leveraging classical intrinsic motivation. |
| Method | Supplied task information | Behavioural evidence | Feedback/refinement |
| WATERFALL | No prescribed goal; restricted code/API | PO visual rollouts; reveal-gated interface | No GT scalar; tournament/history |
| No-images | Same restricted interface | Text digests with gated reveals | No GT scalar; text tournament/history |
| T2R | Task instruction and pre-disclosed schema | No policy rollouts | Success criterion in prompt; one-shot |
| L2R | Instruction, motion descriptor, pre-disclosed schema | No policy rollouts | Success criterion in prompt; two-call generation |
| CAR-VLM | Start/goal images; pre-disclosed compact interface | Goal disclosed visually; no policy rollouts | No scalar fitness; one-shot |
| EUREKA | Goal description and schema | No policy rollouts | Success/length/reward statistics; reflection |
| Category | Setting |
| Hardware | AMD Ryzen 9 5950X; 128GB RAM; NVIDIA RTX 3090 24GB |
| Operating System | Ubuntu 24.04 LTS |
| Simulator | Unity3D 2022.3 LTS |
| RL Toolkit | Unity ML-Agents 0.30.0 with PyTorch backend |
| Core RL optimiser | PPO for WATERFALL candidates and RL baselines |
| Additional RL Modules | LSTM, ICM, RND, and CBE where specified |
| Component | Time (h) | API calls | Tokens (M) | API cost (USD) |
| Stage I: generation | 0.59 | 11 | 0.12 | 1.12 |
| PPO training | 18.00 | – | – | – |
| Stage II: tournaments | 0.72 | 51 | 4.06 | 8.48 |
| Stage III: assessment/mutation | 0.64 | 23 | 0.71 | 2.95 |
| Accounted total | 19.95 | 85 | 4.89 | 12.55 |
| Setting | Env-steps (B) | PPO wall-clock (h) | PPO stream-h | VLM wall-clock (h) | VLM cost (USD) | VLM tokens (M) |
| I FO | 0.71 | 10.27 | 721 | 2.30 | 20.82 | 5.68 |
| I PO | 1.00 | 19.04 | 1,668 | 3.44 | 25.11 | 6.28 |
| II FO | 3.40 | 47.72 | 3,193 | 6.15 | 54.12 | 14.56 |
| II PO | 2.60 | 64.20 | 4,233 | 6.61 | 52.05 | 13.25 |
| III FO | 1.00 | 15.79 | 1,541 | 2.53 | 15.89 | 6.23 |
| III PO | 1.00 | 18.00 | 1,601 | 1.95 | 12.55 | 4.89 |
| Method | Steps (B) | PPO stream-h | API cost (USD) |
| WATERFALL | 9.89 | 20,579 | 227.51 |
| CAR-VLM | 45.64 | 92,220 | 19.20 |
| EUREKA | 44.69 | 90,589 | 95.60 |
| L2R | 42.82 | 86,997 | 17.38 |
| T2R | 27.84 | 63,866 | 20.39 |
| Sparse PPO | 6.18 | 10,219 | – |
| PPO Hyperparameter Settings | ||||
| Hyperparameter | Env. I | Env. II | Env. III | Env. IV |
| Minibatch size | 2048 | 2048 | 2048 | 4096 |
| Rollout buffer size | 20480 | 20480 | 20480 | 40960 |
| Learning rate | ||||
| Learning-rate schedule | Linear | Linear | Linear | Linear |
| Number of epochs | 3 | 3 | 3 | 3 |
| Intrinsic-Motivation Settings | ||
| Hyperparameter | RL+ICM | RL+RND |
| Auxiliary reward strength | 0.1 | 0.1 |
| Auxiliary discount factor | 0.999 | 0.999 |
| Auxiliary learning rate | ||
| Auxiliary hidden layers | 2 | 2 |
| Auxiliary hidden units per layer | 128 | 128 |
| Policy | PRM regime | Scored/attempted | MCC | FPR | Precision |
| Random | Memory | 87/87 | 0.211 | 32.9% | 6.7% |
| Random | Privileged | 87/87 | 0.229 | 29.4% | 7.4% |
| Random | Vision | 87/87 | 0.174 | 42.4% | 5.3% |
| 1M | Memory | 63/65 | 0.102 | 75.5% | 24.5% |
| 1M | Privileged | 65/65 | 0.031 | 70.0% | 23.9% |
| 1M | Vision | 65/65 | 0.143 | 72.0% | 26.5% |
| Call No. | Environment | Candidate | STOP (M steps) | Seeds solved | Completion range (%) | |
| Solved seeds | Below threshold | |||||
| No. #1 | Env I FO | k1 | 10 | 10/10 | 77.7–86.8 | — |
| No. #2 | Env I FO | k7 | 10 | 10/10 | 78.8–84.6 | — |
| No. #3 | Env I PO | k1 | 10 | 10/10 | 94.7–99.6 | — |
| No. #4 | Env I PO | k6 | 10 | 10/10 | 66.1–72.9 | — |
| No. #5 | Env I PO | k8 | 10 | 9/10 | 66.8–92.4 | 23.0 |
| Evaluator | Cost/run | Tok./run | Count FP/Miss/U | EvMiss | |||||
| GPT-5 Nano | 2.28M | 0/3 | 0/3 | 0/3 | 0/3 | 0.04/0.22/0.86 | 0 | ||
| GPT-4.1 Nano | 3.50M | 0/3 | 0/3 | 0/3 | 0/3 | 0.19/0.16/0.40 | 14 | ||
| GPT-5 Mini | 1.71M | 3/3 | 3/3 | 3/3 | 3/3 | 0.23/0.02/0.11 | 0 | ||
| GPT-4.1 Mini | 2.29M | 0/3 | 0/3 | 0/3 | 0/3 | 0.41/0.08/0.06 | 2 | ||
| GPT-4.1 | 1.10M | 3/3 | 3/3 | 3/3 | 2/3 | 0.09/0.00/0.00 | 0 | ||
| GPT-5.4 Nano-L | 1.76M | 2/3 | 0/3 | 0/3 | 0/3 | 0.00/0.25/0.04 | 0 |
| Checkpoint M | Checkpoint M | Total (both checkpoints) | ||||||||
| Topology | Top-1 | Top-1 | Matches | Requests | Cost | |||||
| KoH | \2.97$ | |||||||||
| RR | – | – | \15.40$ | |||||||
| Swiss | \8.48$ | |||||||||
| Study | Setting | Runs | Non-tied | Accuracy | Brier | Both elites | |
| Episode count | 30 | 462 | 83.5% | 0.738 | 0.142 | 10/30 | |
| 12 | 180 | 93.9% | 0.846 | 0.081 | 9/12 | ||
| 9 | 129 | 89.9% | 0.797 | 0.083 | 9/9 | ||
| 3 | 46 | 97.8% | 0.869 | 0.044 | 3/3 | ||
| Temporal stride | 3 | 49 | 93.9% | 0.765 | 0.085 | 1/3 | |
| 3 | 48 | 100.0% | 0.865 | 0.057 | 3/3 |
| Resolution | Runs | Cost/run | Total cost | Pattern | ||||
| 4096px | 1 | \5.155$ | \5.16$ | Reference | ||||
| 1024px | 3 | \3.926\pm 0.124$ | \11.78$ | Invariant | ||||
| 768px | 3 | \2.510\pm 0.183$ | \7.53$ | Invariant | ||||
| 512px | 3 | \1.658\pm 0.020$ | \4.97$ | Invariant | ||||
| 256px | 3 | \1.358\pm 0.052$ | \4.07$ | Retains only | ||||
| 128px | 3 | \1.099\pm 0.098$ | \3.30$ | Unstable |
| Res. | Run | Top-1 | Cost | |||||
| 4096 | 1 | Y | Y | Y | \5.155$ | |||
| 1024 | 1 | Y | Y | Y | \3.785$ | |||
| 1024 | 2 | Y | Y | Y | \4.017$ | |||
| 1024 | 3 | Y | Y | Y | \3.977$ | |||
| 768 | 1 | Y | Y | Y | \2.487$ | |||
| 768 | 2 | Y | Y | Y | \2.703$ |
| Stride | Runs | Cost/run | Total cost | Top-1 runs | ||||
| 3 | \4.737\pm 0.054$ | \14.21$ | ||||||
| 3 | \2.948\pm 0.153$ | \8.84$ | ||||||
| 3 | \1.955\pm 0.039$ | \5.87$ | ||||||
| 3 | \1.451\pm 0.037$ | \4.35$ | ||||||
| 3 | \1.259\pm 0.042$ | \3.78$ | ||||||
| 3 | \1.131\pm 0.019$ | \3.39$ |
| Stride | Run | Top-1 | Cost | ||||||
| 1 | Y | N | N | N | \4.798$ | ||||
| 2 | Y | N | N | N | \4.718$ | ||||
| 3 | Y | Y | Y | N | \4.694$ | ||||
| 1 | Y | Y | Y | N | \2.790$ | ||||
| 2 | Y | Y | Y | N | \3.096$ | ||||
| 3 | Y | Y | Y | N | \2.958$ |
| Subsets | Judge calls | Cost/run | |||||
| 10 | 30 | \1.015\pm 0.080$ | |||||
| 4 | 12 | \2.131\pm 0.174$ | |||||
| 3 | 9 | \2.911\pm 0.086$ | |||||
| 1 | 3 | \5.242\pm 0.186$ |
| Subset | Episodes | Top-1 across 3 runs | ||||
| 1 | 1 | |||||
| 1 | 2 | |||||
| 1 | 3 | |||||
| 1 | 4 | |||||
| 1 | 5 | |||||
| 1 | 6 |
Reward Design for RL
Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.
See all 290 papers