Learning from reward functions and imitation learning of demonstrations are the two principal approaches for training autonomous systems that interact with an environment through action and observation. Both, however, require human specification for each behaviour to be acquired, a problem for long-lived self-adaptive systems whose goals and operating conditions cannot be fully anticipated at design time. Recently, Goal-Conditioned Supervised Learning (GCSL) through self-imitation has been proposed as a self-supervised alternative: by strategically relabelling goals, agents can derive policy insights from their own experiences. Despite its successes, this framework presents two notable limitations: (1) learning exclusively from self-generated experiences can exacerbate the agents' inherent biases; (2) the relabelling strategy allows agents to focus solely on successful outcomes, precluding them from learning from their mistakes. To address these issues, we propose GCSL with Negative Feedback (GCSL-NF), which evaluates each trajectory twice: positively with respect to relabelled goals, and correctively with respect to the goal originally intended. The corrective target comes from a similarity function learned contrastively from trajectory-induced neighbourhood relations, so that neither a reward function nor a geometric distance needs to be specified. Our experiments show that GCSL-NF overcomes limitations imposed by agents' initial biases, increasingly benefits from negative feedback as learning progresses, and matches or surpasses GCSL- and HER-based methods. By reducing reliance on prespecified reward functions, the proposed approach is particularly relevant for self-adaptive autonomous systems, where adaptation objectives may be diverse, changing, or difficult to engineer.
Figures & tables
Figure 1: A Motivating Example: the agent starts from Point A , initiates its journey with the objective of reaching point B , but ultimately arrives at point C . The green line indicates the actual path, while red dotted lines represent the distance between the achieved state and the original goal.
Figure 2: Diagram of GCSL-NF: The agent learns how to reach goals by sampling trajectories, reconsidering the trajectories with both relabelled goals and original goals, and then updating the policy and similarity function.
Figure 3: Illustration of the learning process for pφ from trajectories. The figure shows how the neighbourhood relationship is defined on trajectories. The vertical axis indicates different trajectories, and the horizontal axis indicates time steps within the same trajectory. The blue region corresponds to positive samples from a unimodal distribution around the reference state, and the red region corresponds to negative samples.
Research question
Evaluated in
Goal-reaching performance
RQ1
Does GCSL-NF match or surpass HER- and GCSL-based baselines on discrete-action goal-reaching tasks, especially when observation-space distance is a poor proxy for physical or temporal proximity (obstacles, bottlenecks, LiDAR)?
Sec. 4.2.1
RQ2
Does GCSL-NF extend effectively to continuous action spaces?
Sec. 4.2.2
Mechanism analysis
RQ3
Is the improvement attributable to combining positive hindsight supervision with original-goal negative feedback, rather than to either signal alone?
Sec. 4.3.1
RQ4
Does the trajectory-induced similarity pφ provide more useful corrective targets than alternative learned distance measures?
Sec. 4.3.2
Table 1: Research questions and corresponding experiments.
Figure 4: Environments: red points indicate the agent position, and green stars denote the goal position.
Figure 5: Discrete-action goal-reaching results across the benchmark environments. GCSL-NF matches or outperforms the baselines on the tested tasks. The results are averaged over 5 random seeds, and the shaded area represents one standard deviation.
Figure 6: Results: Comparison between continuous GCSL-NF, continuous GCSL, HER with DDPG and Contrastive GCRL. (a, b) Position and velocity distance in Car Racing (Point Mass); (d, e) the same in Car Racing (4-Room); (c) Fetch Reach; (f) Fetch Push. The results are averaged over 5 runs with different random seeds. The shaded area represents the standard deviation.
Figure 7: Ablation of positive and negative feedback in the Point Mass Navigation with Obstacles environment. Panel (a) compares policies trained with positive feedback only, negative feedback only, and both feedback signals. Panel (b) shows the relative contribution of the original-goal loss, measured by Lo+L+Lo , over training. Results are averaged over 5 random seeds, and the shaded area represents one standard deviation.
Figure 8: Heatmap: (a) comparison between different kinds of distance measures, and (b) comparison between inverse distance functions learned from different data-collecting policies in the 4-room environment. Each panel shows the values of an inverse distance function with respect to a fixed state, which is indicated by a black star. In (a), the three columns show the inverse distance function proposed by this paper, the distance function learned by SimCLR-TT, and the distance function learned by the successor representation; the first two rows are the results for the 4-room environment, and the last two rows are the results for the LiDAR environment. In (b), the three columns show the inverse distance function learned together with the policy, the inverse distance function learned by trajectories generated by a random policy, and the inverse distance function learned by trajectories generated by an optimal policy; the first two rows are the results for distance functions with threshold n=5 , and the last two rows are for threshold n=25 . Note that darker colour indicates closer proximity, because we have inverse distance measures.
Figure 9: Comparison of learned distance measures and sensitivity to the threshold n , in the 2D 4-Room (a) and 2D LiDAR (b) navigation environments. GCSL-NF is trained with the trajectory-induced similarity pφ at two thresholds ( n=5 , used throughout the main experiments, and n=25 ), with a similarity learned by SimCLR-TT ( pSimCLR-TT ), and with a successor-representation-based similarity ( pSR ), keeping the rest of the algorithm fixed. The two thresholds are essentially indistinguishable in both environments.
Figure 10: Recovery from a corrupted prior and from a runtime goal change. (a) Distance to the goal and (b) action entropy under initial policy bias in the Point Mass Navigation environment, where the policy is pre-corrupted toward rightward movement before training. (c) Re-adaptation in the 2D 4-Room Navigation environment, where the goal distribution shifts to the diagonally opposite room after 20 k episodes (dashed line), without resetting the policy or the replay buffer. Results are averaged over 5 seeds; shaded areas represent one standard deviation.
Figure 11: Learning curves under (a) sensor noise and (b) actuation noise in the 2D LiDAR Navigation environment, at three noise levels ( 2.5% , 5% , 7.5% ). Top row: position distance; bottom row: orientation distance. Results averaged over 5 seeds; shaded area is one standard deviation.
Figure 12: Danger-zone safety generalisation after training-time constraint exposure. Panel (a) compares goal-reaching accuracy and core-zone avoidance across six algorithms and five safety settings. For GCSL-NF, additional radial CDFs over trajectories (b) and steps (c), visit heatmaps (d), and representative trajectories (e) show how different combinations of action-space intervention and negative-feedback clamping affect unsafe-region avoidance. Evaluation is performed with all interventions disabled. Statistics are computed over 5,000 test trajectories per method and setting, across 5 random seeds.
Figure 13: PointmassCorner task and learned behaviours under a history-dependent waypoint rule. (a) Environment schematic. Cross-corner trajectories must pass through a direction-dependent waypoint: TL for BL → TR and BR for TR → BL. (b) Representative trajectories of GCSL-NF and HER DQN under the trajectory-level constraint. Green trajectories correspond to BL → TR trips via TL, and orange trajectories correspond to TR → BL trips via BR. (c) Goal-reaching and waypoint-adherence rates.
Vanilla RL
RL with HER
GCSL
Contrastive GCRL
GCSL-NF
Relabelled goal
✓
✓
✓
✓
Original goal
✓
✓
✓
Contrastive goal
✓
✓
Contrastive action
✓
✓
✓
External reward function
✓
✓
Table 2: Comparison between different algorithms for goal-conditioned problems
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Environment
Obs. dim
Action set
Horizon
Notes
Point Mass Navigation
2
move
50
Obs. range [−1,1] ; step 0.05 with N(0,0.01) noise.
+ Obstacles
2
move
70
Adds a central circular obstacle (radius 0.4 , centre (0,0) ).
2D 4-room Navigation
2
move
70
Obs. range [−1.2,1.2] ; passageways of width 0.4 .
2D LiDAR Navigation
64
move/turn
50
Map 200×200 ; 360∘ LiDAR obs.; step 10 (move), 2π (turn); goals are LiDAR readings at poses (x,y,θ) .
Object Pushing
4
move
50
Obs. range [−0.5,0.5] ; puck side length 0.2 ; obs./goal cover end-effector and puck coordinates.
Appendix
Table 3: Parameters of the five discrete-action environments. Initial states and goals are sampled uniformly. The action set “move” denotes {up, down, left, right, stay}; “move/turn” denotes {forward, backward, turn left, turn right, stay}.
GCSL-NF
GCSL
HER DQN
HER A2C
W-GCSL
Contrastive GCRL
Point Mass Navigation
0.085 ± 0.037
0.187 ± 0.080
0.066 ± 0.019
0.158 ± 0.029
0.127 ± 0.084
0.065 ± 0.014
Point Mass Navigation with Obstacles
0.152 ± 0.015
0.341 ± 0.050
0.299 ± 0.031
0.481 ± 0.105
0.364 ± 0.034
0.213 ± 0.038
2D 4-Room Navigation
0.495 ± 0.089
0.781 ± 0.066
0.658 ± 0.075
1.153 ± 0.312
0.775 ± 0.052
0.668 ± 0.070
2D LiDAR Navigation with Obstacles - Orientation
0.555 ± 0.070
0.749 ± 0.087
1.597 ± 0.086
1.520 ± 0.099
0.654 ± 0.147
1.227 ± 0.096
2D LiDAR Navigation with Obstacles - Position
0.344 ± 0.060
0.435 ± 0.040
0.734 ± 0.030
0.879 ± 0.058
0.379 ± 0.043
0.659 ± 0.033
Object Pushing Puck Position
0.078 ± 0.009
0.306 ± 0.035
0.463 ± 0.005
0.458 ± 0.008
0.334 ± 0.031
0.465 ± 0.019
Appendix
Table 4: Results for baseline algorithms and GCSL-NF
GCSL-NF
GCSL
HER DDPG
Contrastive GCRL
Car Racing Point Mass Navigation without Obstacles
Position
0.043 ± 0.019
0.037 ± 0.010
0.133 ± 0.222
0.173 ± 0.007
Velocity
0.013 ± 0.008
0.016 ± 0.004
0.054 ± 0.023
0.256 ± 0.008
Car Racing Point Mass Navigation with 4-Room
Position
0.387 ± 0.077
1.267 ± 0.045
0.457 ± 0.021
0.296 ± 0.055
Velocity
0.095 ± 0.011
0.233 ± 0.030
0.076 ± 0.004
0.246 ± 0.009
Fetch Reach
0.036 ± 0.003
0.013 ± 0.013
0.079 ± 0.021
0.035 ± 0.002
Fetch Push
0.049 ± 0.014
0.165 ± 0.009
0.171 ± 0.003
0.060 ± 0.007
Appendix
Table 5: Results for baseline algorithms and GCSL-NF for continuous environments