Continual Reinforcement Learning Is an Integration Problem
Organizations: FractalBrain, London, United Kingdom · University College London (UCL), London, United Kingdom · an independent researcher, London, United Kingdom · an independent researcher, Barcelona, Spain ·
Abstract
This paper presents an opinionated set of ten desiderata intended to inform the design and evaluation of reinforcement-learning agents that learn continually throughout a continuing lifetime. We use these desiderata to examine a representative selection of existing approaches. Progress on individual desiderata is valuable, but the longer-term aim should be to integrate all ten within a single agent. To our knowledge, no published evaluation has yet provided evidence for such an integration. Likewise, the representative benchmarks we review probe different parts of this target, but separate evaluations do not establish that the corresponding properties coexist within one agent throughout a continuing lifetime. As a secondary contribution, we therefore present a compact persistent-world scenario as a playable companion to the desiderata and outline a proposed benchmark that examines a broad range of associated behavioral capabilities together within one such lifetime.
Figures & tables
| Flavor | Immediate effect | Delayed effect | Functional role |
|---|---|---|---|
| Satiety | Increases satiety by a modest amount. | None. | Supports routine satiety maintenance as satiety decays. |
| Hydration | Increases hydration by an amount calibrated to offset the delayed hydration loss from one salty bean. | None. | Supports routine hydration maintenance as hydration decays and provides prepared recovery after salty-bean use. |
| Bitter | Applies a reward penalty. | None. | Makes indiscriminate consumption costly, rewarding agents that learn and use color–effect associations. |
| Salty | Increases satiety; this benefit is calibrated independently of the later hydration loss. | Decreases hydration after a fixed delay. | Creates a useful but risky action: exploit it when hydration recovery is prepared and avoid it otherwise. |
| Setting | Suggested default |
|---|---|
| Field of view | cells |
| Inventory capacity and initial state | bean; empty |
| Color variants per flavor | |
| Bean probability per cell | |
| Color-biome side length | cells |
| Resource range |
| Lifetime situation | Associated readouts | |
|---|---|---|
| D2 | Depletion of non-regrowing local beans and crossings among biomes that expose unfamiliar or recurring color variants after intervening experience. | Acquisition of new mappings, retained color–flavor knowledge when variants recur, forward transfer, and late-life plasticity. |
| D6 & D7 | Early observation of resource-changing bean effects while both resources remain nonnegative, followed much later by the first endogenous deficit. The salty bean adds a short action–consequence separation, while inventory use and the optional fence introduce additional memory demands. | First corrective action, appropriateness of the first bean eaten, steps and cumulative deficit until recovery, and timely use of a stored hydration bean after the delayed loss, stratified by prior exposure and elapsed interval; separate summaries for the short-delay and memory-dependent situations. |
| D8 | First encounters with previously unseen salty-bean configurations after the agent has separately experienced salty- and hydration-bean effects, with recovery either already prepared or requiring additional search. | First-encounter conditional salty-bean use, stratified by prior exposure to the component effects; exploiting prepared opportunities while rejecting unprepared ones provides evidence for composition beyond retrieval of an exact earlier episode. |
| D9 | A deterministic immediate satiety benefit followed by a delayed hydration loss that requires prepared recovery, while both resources decay and the blue-light objective continues. | Resource-wise outcomes, frequency and cost of unprepared salty-bean use, blue-light reward forgone during preparation and recovery, and state-conditional quality of recovery preparation. |
| D10 | A long-term objective of tracking moving regions of strong blue light combined with short-term resource maintenance and intermediate preparation for salty-bean use. | Blue-light tracking while resources are safe, appropriate interruption for maintenance, successful preparation, and resumption of blue-light pursuit. |
Continual RL
Continual reinforcement learning studies how agents adapt to changing tasks while retaining skills learned earlier.
See all 58 papers