cs.LGNov 17, 2026

WATERFALL: Workflow for Adaptive Training with Evolutionary Reward Formulation and Automated Learning Loops

Authors: Eleftherios Triantafyllidis, Filippos Christianos, Zhibin Li, Bernd Bickel

Organizations: ETH Zurich, Zurich, Switzerland · UCL, London, UK

Abstract

Reward design remains a central bottleneck in reinforcement learning (RL), particularly in sparse-reward, long-horizon and partially observable settings requiring dependency chaining, memory, and deceptive affordance disambiguation. While recent foundation-model approaches reduce manual reward engineering, most assume goal-conditioning, task descriptions, expert demonstrations, or ground-truth metrics. These assumptions are problematic in partially observable environments, where task-relevant objectives, affordances, and dynamics must be discovered through interaction rather than disclosed a priori. We formalise this problem setting by introducing WATERFALL, an iterative population-based workflow for automated reward discovery under the strict observability constraints inherent in POMDP environments. WATERFALL achieves this by (i) synthesising diverse programmatic reward candidates via persona-conditioned generation, (ii) evaluating candidate behaviour from raw visual rollouts utilising a Swiss-system evolutionary tournament judged by vision-language models to identify elites, and (iii) leveraging longitudinal assessment histories to iteratively mutate, escalate, or prune candidates. We evaluate WATERFALL across fully observable and reveal-gated partially observable environments ranging from discrete to continuous, comparing against sparse-reward RL, intrinsic-motivation and foundation-model reward-synthesis baselines. Our results indicate that WATERFALL discovers and refines reward programs without access to privileged information, outperforming baselines that retain their native information advantages, most clearly in the POMDP regime. Extensive ablations substantiate that visual evidence, tournament-based selection, and iterative refinement are jointly important. Enforcing this contract is what separates autonomous reward discovery from benchmark-privileged reward shaping.

Figures & tables

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Specifying Reward Functions for RL Without Environment Sampling

    Sep 14, 2026Stephane Hatgis-Kessell, W. Bradley Knox, Emma BrunskillLLM-Guided RLReward Design for RL

  2. R2E:R^2E: A Role-driven Reward Evolutionary Framework for Automated Reward Function Design

    Oct 23, 2026Shouhao Chang, Xuan Liu, Hongye Zhu +2Reward Design for RLLLM-Guided RL

  3. Dual-Evolving Reinforcement Learning for Automated Reward Design

    Dec 15, 2025Sitao Cheng, Tianle Li, Xuhan Huang +2Reward Design for RLReinforcement Learning