Learning from reward functions and imitation learning of demonstrations are the two principal approaches for training autonomous systems that interact with an environment through action and observation. Both, however, require human specification for each behaviour to be acquired, a problem for long-lived self-adaptive systems whose goals and operating conditions cannot be fully anticipated at design time. Recently, Goal-Conditioned Supervised Learning (GCSL) through self-imitation has been proposed as a self-supervised alternative: by strategically relabelling goals, agents can derive policy insights from their own experiences. Despite its successes, this framework presents two notable limitations: (1) learning exclusively from self-generated experiences can exacerbate the agents' inherent biases; (2) the relabelling strategy allows agents to focus solely on successful outcomes, precluding them from learning from their mistakes. To address these issues, we propose GCSL with Negative Feedback (GCSL-NF), which evaluates each trajectory twice: positively with respect to relabelled goals, and correctively with respect to the goal originally intended. The corrective target comes from a similarity function learned contrastively from trajectory-induced neighbourhood relations, so that neither a reward function nor a geometric distance needs to be specified. Our experiments show that GCSL-NF overcomes limitations imposed by agents' initial biases, increasingly benefits from negative feedback as learning progresses, and matches or surpasses GCSL- and HER-based methods. By reducing reliance on prespecified reward functions, the proposed approach is particularly relevant for self-adaptive autonomous systems, where adaptation objectives may be diverse, changing, or difficult to engineer.
Figures & tables
Figure 1: A Motivating Example: the agent starts from Point A , initiates its journey with the objective of reaching point B , but ultimately arrives at point C . The green line indicates the actual path, while red dotted lines represent the distance between the achieved state and the original goal.
Figure 2: Diagram of GCSL-NF: The agent learns how to reach goals by sampling trajectories, reconsidering the trajectories with both relabelled goals and original goals, and then updating the policy and similarity function.
Figure 3: Illustration of the learning process for pφ from trajectories. The figure shows how the neighbourhood relationship is defined on trajectories. The vertical axis indicates different trajectories, and the horizontal axis indicates time steps within the same trajectory. The blue region corresponds to positive samples from a unimodal distribution around the reference state, and the red region corresponds to negative samples.
Research question
Evaluated in
Goal-reaching performance
RQ1
Does GCSL-NF match or surpass HER- and GCSL-based baselines on discrete-action goal-reaching tasks, especially when observation-space distance is a poor proxy for physical or temporal proximity (obstacles, bottlenecks, LiDAR)?
Sec. 4.2.1
RQ2
Does GCSL-NF extend effectively to continuous action spaces?
Sec. 4.2.2
Mechanism analysis
RQ3
Is the improvement attributable to combining positive hindsight supervision with original-goal negative feedback, rather than to either signal alone?
Sec. 4.3.1
RQ4
Does the trajectory-induced similarity pφ provide more useful corrective targets than alternative learned distance measures?
Sec. 4.3.2
Table 1: Research questions and corresponding experiments.
Figure 4: Environments: red points indicate the agent position, and green stars denote the goal position.
Figure 5: Discrete-action goal-reaching results across the benchmark environments. GCSL-NF matches or outperforms the baselines on the tested tasks. The results are averaged over 5 random seeds, and the shaded area represents one standard deviation.
Figure 6: Results: Comparison between continuous GCSL-NF, continuous GCSL, HER with DDPG and Contrastive GCRL. (a, b) Position and velocity distance in Car Racing (Point Mass); (d, e) the same in Car Racing (4-Room); (c) Fetch Reach; (f) Fetch Push. The results are averaged over 5 runs with different random seeds. The shaded area represents the standard deviation.
Figure 7: Ablation of positive and negative feedback in the Point Mass Navigation with Obstacles environment. Panel (a) compares policies trained with positive feedback only, negative feedback only, and both feedback signals. Panel (b) shows the relative contribution of the original-goal loss, measured by Lo+L+Lo , over training. Results are averaged over 5 random seeds, and the shaded area represents one standard deviation.
Figure 8: Heatmap: (a) comparison between different kinds of distance measures, and (b) comparison between inverse distance functions learned from different data-collecting policies in the 4-room environment. Each panel shows the values of an inverse distance function with respect to a fixed state, which is indicated by a black star. In (a), the three columns show the inverse distance function proposed by this paper, the distance function learned by SimCLR-TT, and the distance function learned by the successor representation; the first two rows are the results for the 4-room environment, and the last two rows are the results for the LiDAR environment. In (b), the three columns show the inverse distance function learned together with the policy, the inverse distance function learned by trajectories generated by a random policy, and the inverse distance function learned by trajectories generated by an optimal policy; the first two rows are the results for distance functions with threshold n=5 , and the last two rows are for threshold n=25 . Note that darker colour indicates closer proximity, because we have inverse distance measures.
Figure 9: Comparison of learned distance measures and sensitivity to the threshold n , in the 2D 4-Room (a) and 2D LiDAR (b) navigation environments. GCSL-NF is trained with the trajectory-induced similarity pφ at two thresholds ( n=5 , used throughout the main experiments, and n=25 ), with a similarity learned by SimCLR-TT ( pSimCLR-TT ), and with a successor-representation-based similarity ( pSR ), keeping the rest of the algorithm fixed. The two thresholds are essentially indistinguishable in both environments.
Figure 10: Recovery from a corrupted prior and from a runtime goal change. (a) Distance to the goal and (b) action entropy under initial policy bias in the Point Mass Navigation environment, where the policy is pre-corrupted toward rightward movement before training. (c) Re-adaptation in the 2D 4-Room Navigation environment, where the goal distribution shifts to the diagonally opposite room after 20 k episodes (dashed line), without resetting the policy or the replay buffer. Results are averaged over 5 seeds; shaded areas represent one standard deviation.
Figure 11: Learning curves under (a) sensor noise and (b) actuation noise in the 2D LiDAR Navigation environment, at three noise levels ( 2.5% , 5% , 7.5% ). Top row: position distance; bottom row: orientation distance. Results averaged over 5 seeds; shaded area is one standard deviation.
Figure 12: Danger-zone safety generalisation after training-time constraint exposure. Panel (a) compares goal-reaching accuracy and core-zone avoidance across six algorithms and five safety settings. For GCSL-NF, additional radial CDFs over trajectories (b) and steps (c), visit heatmaps (d), and representative trajectories (e) show how different combinations of action-space intervention and negative-feedback clamping affect unsafe-region avoidance. Evaluation is performed with all interventions disabled. Statistics are computed over 5,000 test trajectories per method and setting, across 5 random seeds.
Figure 13: PointmassCorner task and learned behaviours under a history-dependent waypoint rule. (a) Environment schematic. Cross-corner trajectories must pass through a direction-dependent waypoint: TL for BL → TR and BR for TR → BL. (b) Representative trajectories of GCSL-NF and HER DQN under the trajectory-level constraint. Green trajectories correspond to BL → TR trips via TL, and orange trajectories correspond to TR → BL trips via BR. (c) Goal-reaching and waypoint-adherence rates.
Vanilla RL
RL with HER
GCSL
Contrastive GCRL
GCSL-NF
Relabelled goal
✓
✓
✓
✓
Original goal
✓
✓
✓
Contrastive goal
✓
✓
Contrastive action
✓
✓
✓
External reward function
✓
✓
Table 2: Comparison between different algorithms for goal-conditioned problems
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Environment
Obs. dim
Action set
Horizon
Notes
Point Mass Navigation
2
move
50
Obs. range [−1,1] ; step 0.05 with N(0,0.01) noise.
+ Obstacles
2
move
70
Adds a central circular obstacle (radius 0.4 , centre (0,0) ).
2D 4-room Navigation
2
move
70
Obs. range [−1.2,1.2] ; passageways of width 0.4 .
2D LiDAR Navigation
64
move/turn
50
Map 200×200 ; 360∘ LiDAR obs.; step 10 (move), 2π (turn); goals are LiDAR readings at poses (x,y,θ) .
Object Pushing
4
move
50
Obs. range [−0.5,0.5] ; puck side length 0.2 ; obs./goal cover end-effector and puck coordinates.
Appendix
Table 3: Parameters of the five discrete-action environments. Initial states and goals are sampled uniformly. The action set “move” denotes {up, down, left, right, stay}; “move/turn” denotes {forward, backward, turn left, turn right, stay}.
GCSL-NF
GCSL
HER DQN
HER A2C
W-GCSL
Contrastive GCRL
Point Mass Navigation
0.085 ± 0.037
0.187 ± 0.080
0.066 ± 0.019
0.158 ± 0.029
0.127 ± 0.084
0.065 ± 0.014
Point Mass Navigation with Obstacles
0.152 ± 0.015
0.341 ± 0.050
0.299 ± 0.031
0.481 ± 0.105
0.364 ± 0.034
0.213 ± 0.038
2D 4-Room Navigation
0.495 ± 0.089
0.781 ± 0.066
0.658 ± 0.075
1.153 ± 0.312
0.775 ± 0.052
0.668 ± 0.070
2D LiDAR Navigation with Obstacles - Orientation
0.555 ± 0.070
0.749 ± 0.087
1.597 ± 0.086
1.520 ± 0.099
0.654 ± 0.147
1.227 ± 0.096
2D LiDAR Navigation with Obstacles - Position
0.344 ± 0.060
0.435 ± 0.040
0.734 ± 0.030
0.879 ± 0.058
0.379 ± 0.043
0.659 ± 0.033
Object Pushing Puck Position
0.078 ± 0.009
0.306 ± 0.035
0.463 ± 0.005
0.458 ± 0.008
0.334 ± 0.031
0.465 ± 0.019
Appendix
Table 4: Results for baseline algorithms and GCSL-NF
GCSL-NF
GCSL
HER DDPG
Contrastive GCRL
Car Racing Point Mass Navigation without Obstacles
Position
0.043 ± 0.019
0.037 ± 0.010
0.133 ± 0.222
0.173 ± 0.007
Velocity
0.013 ± 0.008
0.016 ± 0.004
0.054 ± 0.023
0.256 ± 0.008
Car Racing Point Mass Navigation with 4-Room
Position
0.387 ± 0.077
1.267 ± 0.045
0.457 ± 0.021
0.296 ± 0.055
Velocity
0.095 ± 0.011
0.233 ± 0.030
0.076 ± 0.004
0.246 ± 0.009
Fetch Reach
0.036 ± 0.003
0.013 ± 0.013
0.079 ± 0.021
0.035 ± 0.002
Fetch Push
0.049 ± 0.014
0.165 ± 0.009
0.171 ± 0.003
0.060 ± 0.007
Appendix
Table 5: Results for baseline algorithms and GCSL-NF for continuous environments
Self-evolving agents improve their performance on long-horizon tasks by learning from their own interactions with an environment. A common approach uses the resulting failed trajectories as negatives for preference training. However, collecting an agent's own failures does not ensure that they provide informative supervision for further improvement. Obvious failures may be easy to reject without learning to identify errors in more plausible attempts. These plausible but incorrect trajectories can serve as hard negatives, helping agents learn to distinguish successful behavior from failures with less obvious errors. To enable agents to generate and learn from such informative negatives, we propose a co-evolving framework that couples a target agent's self-improvement with a failure agent trained to generate hard negatives. The failure agent learns exclusively from both agents' failed trajectories to generate higher-reward failures, which the target agent uses as negatives for preference training. As the target policy improves, its failures provide new training data for the failure agent. The updated failure agent, in turn, provides new negatives for target training, allowing negative generation to adapt alongside the target policy. Across online shopping, scientific reasoning, and interactive SQL querying, our framework improves average task reward by 5.7 over the baseline. These results demonstrate the value of learning to generate and use informative failures for agent self-improvement. We will make our code publicly available.
Yeonsung Jung, Trilok Padhi, Sina Shaham +4
KAIST · Georgia State University · Independent Researcher +2
Self-training systems often degenerate due to the lack of an external criterion for judging data quality, leading to reward hacking and semantic drift. This paper provides a proof-of-concept system architecture for stable self-training under sparse external feedback and bounded memory, and empirically characterises its learning dynamics and failure modes. We introduce a self-training architecture in which learning is mediated exclusively by environmental viability, rather than by reward, objective functions, or externally defined fitness criteria. Candidate behaviours are executed under real resource constraints, and only those whose environmental effects both persist and preserve the possibility of future interaction are propagated. The environment does not provide semantic feedback, dense rewards, or task-specific supervision; selection operates solely through differential survival of behaviours as world-altering events, making proxy optimisation impossible and rendering reward-hacking evolutionarily unstable. Analysis of semantic dynamics shows that improvement arises primarily through the persistence of effective and repeatable strategies under a regime of consolidation and pruning, a paradigm we refer to as negative-space learning (NSL), and that models develop meta-learning strategies (such as deliberate experimental failure in order to elicit informative error messages) without explicit instruction. This work establishes that environment-grounded selection enables sustainable open-ended self-improvement, offering a viable path toward more robust and generalisable autonomous systems without reliance on human-curated data or complex reward shaping.
Jennifer Dodgson, Alfath Daryl Alhajir, Michael Joedhitya +6
Zero-shot imitation learning requires an agent to reproduce expert behavior from a single demonstration without additional environment interaction or gradient updates at test time. We introduce Contrastive Inverse Reinforcement Learning (CIRL), a self-supervised framework for pre-training zero-shot imitation agents. Our methods rests on a key observation that many useful tasks can be summarized by a single goal state. We can thus convert the multi-task inverse RL problem into a more tractable goal-inference problem, and utilize state-of-the-art goal-conditioned RL methods to recover a policy that reaches the goal. During pre-training, CIRL jointly employs three components to learn without any rewards or demonstrations: (1) a variant of contrastive RL designed to learn maximum-entropy goal-conditioned policies, (2) an automatic goal proposal mechanism (GoalKDE) that drives exploration, and (3) a mean-field variational model that performs amortized goal inference from trajectories. We prove that this procedure consistently recovers the demonstrator's intent by accounting for the relative difficulty of reaching different states and show how structurally similar prior work may otherwise fail to infer the correct reward. Experiments on goal-conditioned and standard reward-maximizing control tasks show that CIRL outperforms prior zero-shot imitation methods, supporting the expressiveness of goals as a compact summary of behavior.
Kathryn Wantlin, Chongyi Zheng, Benjamin Eysenbach
Department of Computer Science, Princeton University, Princeton, NJ, USA.