Collision-free reaching is central to manipulation, yet real cells often leave the size and exact position of an obstacle unobserved, which excludes reinforcement learning controllers that take obstacle geometry as input. A reward limited to a collision penalty does not depend on the clearance outside the obstacle, so under a step cost its learned policy skims the obstacle surface and fails once the obstacle radius exceeds its training value. We investigate whether a single analytic reward term can induce a safety margin without exposing the obstacle to the policy. We augment the reward of a simulated Franka Emika Panda reaching task with a horizontal artificial potential field that penalizes proximity within a safety margin around each of two pillars (fixed nominal layout, random drift). Under hidden observation, policies trained with this term maintain a safe success rate above 88% across a radius sweep from 0.03 m to 0.055 m (30 seeds), while policies without it fall below 10% at the maximum tested radius; the term induces a minimum clearance of 47.9 mm against 9.6 mm without it and eliminates training collisions at least eight times earlier (median steps). The advantage is statistically significant at all out-of-distribution radii and holds across three reinforcement learning solvers and under full observation. Its safe success rate is comparable to that of a classical potential-field planner given the pillar positions. The margin is a fixed buffer rather than an adaptation to the radius, and it lengthens the path by about 19%.
Figures & tables
Figure 1: Hidden-observation training loop. The policy maps the 9 -dimensional state (end-effector position and velocity, goal) to a 3 -dimensional action; the pillars are absent from the state. Obstacle information reaches the policy only through the reward: the collision penalty and the safety augmentation (horizontal potential Φ and height penalty H ).
Figure 2: Observation model. Top: workspace (top view); the dark polyline denotes the schematic manipulator from its base to the end-effector at p . Bottom: observation vector received by the policy, with segment widths equal to the corresponding dimensions. (a) Hidden observation ( R9 ): the policy receives p , p˙ and g ; the pillars are withheld. (b) Full observation ( R9+4N ): the per-pillar features fi=[ci,x−x,ci,y−y,dc,i,h−z] are appended.
Figure 3: Artificial potential field around one pillar. (a) The horizontal potential Φ (red shading) acts within the height band 0≤z≤h , and the height penalty H (blue shading) acts above the pillar top; the dashed circle on the floor marks the outer boundary r=ρ+dsafe of Φ . (b) Φ as a function of the horizontal distance r from the pillar center: it equals one inside the pillar and decreases quadratically to zero at r=ρ+dsafe . (c) H as a function of the height z .
Term
Meaning
Default value
−dt
dense distance to goal
1 (native)
λb1[dt<δ]
success bonus
5.0
λpmax(0,dt−1−dt)
goal-progress reward
1.0
−λσσt
action-magnitude smoothness
0.03
−λsΦ(pt)
horizontal APF potential
1.5
−λhH(pt)
over-the-top height penalty
2.0
Table 1: Reward terms and their default coefficients in Equation ( 9 ).
Configuration
Removed terms
ours
none
no_apf
λsΦ(pt)+λhH(pt)
no_progress
λpmax(0,dt−1−dt)
vanilla
all auxiliary terms
no_xy_apf
λsΦ(pt)
Table 2: Reward configurations obtained by removing terms from Equation ( 9 ).
Figure 4: Rendered experimental scene (PyBullet, panda-gym PandaReach-v3 ). The end-effector starts at the position marked “Start (EE)” and is required to reach a goal sampled inside the orange box (the orange sphere is one example goal), passing the two staggered pillars.
Figure 5: Task geometry at the training radius ( ρ=0.03m ). (a) Top view: the direct line from the start to the example goal passes within ρ of both pillar centers. (b) Side view (projection onto the xz plane): the pillars are 0.21m high, and all goals are sampled below the pillar top. Dashed outlines show the range of pillar-center motion ( ±15mm along x ); the shaded regions indicate the goal sampling area in (a) and the goal height range in (b).
Figure 6: Evaluation radius sweep. (a) Pillar outlines for the six evaluation radii (top view, nominal pillar centers). (b) Gap between the two pillar surfaces as a function of ρ at the nominal centers (markers: the evaluation radii); the corridor closes at ρ≈0.062m .
Group
Obstacle radius ρ (m)
0.030
0.035
0.040
0.045
0.050
0.055
ours
94.3
94.3
94.3
94.3
93.5
89.4
no_apf
92.5
69.3
32.9
22.8
17.2
9.5
no_progress
93.7
93.7
93.7
93.5
91.8
88.3
vanilla
95.3
68.1
30.2
22.7
17.2
9.7
no_xy_apf
83.6
64.6
39.8
27.7
16.6
7.5
Table 3: Safe success rate (%) under hidden observation as a function of obstacle radius. Mean over n=30 seeds per group, each evaluated on 50 goals.
Figure 7: (a) Collision rate, (b) safe success rate, (c) trajectory length, and (d) minimum clearance vs. obstacle radius under hidden observation ( n=30 seeds per group; error bars denote 95% confidence intervals).
Group
Clearance dc (mm)
Center distance dc+ρ (mm)
ours
47.9±11.8
77.9
no_apf
9.6±5.9
39.6
no_progress
45.8±8.1
75.8
vanilla
9.7±6.8
39.7
no_xy_apf
8.4±8.0
38.4
Table 4: Minimum horizontal clearance and inferred center distance at the training radius ρ=0.03m (mean ± sample standard deviation over n=30 seeds per group).
Figure 8: Minimum distance from the pillar center vs. obstacle radius ( n=30 seeds per group; error bars denote 95% confidence intervals). The dotted diagonal marks the pillar surface, i.e., zero clearance; the curves of no_apf and vanilla coincide.
Figure 9: Deterministic rollouts of one trained policy per group (PPO, seed 1) toward the example goal at the training radius ρ=0.03m . (a) Top view. (b) Side view (projection onto the xz plane). Pillars are drawn at their nominal centers.
Reward
Penalty
n
Clearance (mm)
Safe success at ρ (m)
Seeds ≥50%
0.030
0.040
0.055
at 0.055
no_apf
0.2
30
9.6±5.9
92.5
32.9
9.5
1/30
no_apf
5
10
9.9±8.6
94.8
24.4
8.8
1/10
no_apf
10
10
17.3±11.4
90.2
55.6
18.0
2/10
ours
0.2
30
47.9±11.8
94.3
94.3
89.4
30/30
Table 5: Effect of the collision penalty on the no_apf reward (PPO, hidden observation). Clearance is the minimum horizontal clearance at the training radius ρ=0.03m (mean ± sample standard deviation over seeds); safe success rate in %. The last column counts the seeds whose safe success rate at ρ=0.055m is at least 50% .
Figure 10: Effect of the collision penalty on the no_apf reward (hidden observation, PPO). (a) Safe success rate and (b) minimum clearance vs. obstacle radius. Lines show the mean over seeds ( n=30 for the default penalty and for ours , n=10 for penalties 5 and 10 ), error bars denote 95% confidence intervals, and faint points are the individual seeds.
Figure 11: Learning curves: (a) collision rate and (b) safe success rate vs. training steps ( n=30 seeds per group, 20 evaluation goals per checkpoint; error bars denote 95% confidence intervals). The APF groups reach zero collision at a median of 1.5×104 steps, whereas no_apf reaches it at a median of 1.2×105 steps among the 21 converging seeds, and 9/30 seeds do not converge within the 1.5×105 -step budget.
no_apf
ours
ρ (m)
Mean (%)
95% CI
Mean (%)
95% CI
p
0.030
92.5
[88.5,96.5]
94.3
[91.7,96.8]
0.44 (n.s.)
0.035
69.3
[62.5,76.0]
94.3
[91.7,96.8]
2.2×10−8
0.040
32.9
[20.7,45.0]
94.3
[91.7,96.8]
2.0×10−11
0.045
22.8
[11.7,33.9]
94.3
[91.7,96.8]
4.0×10−14
0.050
17.2
[8.1,26.3]
93.5
[90.8,96.3]
9.8×10−18
Table 6: Safe success rate: 95% confidence interval and two-sided Welch t -test ( ours vs. no_apf , n=30 each).
Figure 12: Safe success rate (a) and minimum clearance (b) vs. obstacle radius for ours , no_apf , and no_xy_apf ( n=30 seeds per group; error bars denote 95% confidence intervals).
Obstacle radius ρ (m)
Method
0.030
0.035
0.040
0.045
0.050
0.055
Classical APF (sees pillars)
90.0
90.0
90.0
90.0
90.0
90.0
RL ours
94.3
94.3
94.3
94.3
93.5
89.4
RL no_apf
92.5
69.3
32.9
22.8
17.2
9.5
Table 7: Safe success rate (%) of the classical APF planner (deterministic, 50 goals, full pillar state) and of the RL policies (hidden observation, mean over n=30 seeds).
Figure 13: (a) Safe success rate and (b) collision rate vs. obstacle radius for the classical APF planner (full pillar state, 50 goals, deterministic) and the RL policies (hidden observation, n=30 seeds; error bars denote 95% confidence intervals).
Figure 14: Collision rate vs. obstacle radius under full observation ( n=20 seeds) and hidden observation ( n=30 seeds). Error bars denote 95% confidence intervals.
Solver
Group
Obstacle radius ρ (m)
0.030
0.035
0.040
0.045
0.050
0.055
SAC
ours
85.5
85.4
85.1
85.1
85.1
85.1
SAC
no_apf
99.2
70.5
18.1
10.4
7.5
2.9
TD3
ours
88.7
88.7
88.7
88.7
88.2
85.5
TD3
no_apf
100.0
68.2
18.3
14.4
12.3
5.1
Table 8: Safe success rate (%) for SAC and TD3 ( n=20 seeds each).
Figure 15: Safe success rate vs. obstacle radius for (a) PPO, (b) SAC, and (c) TD3, each panel comparing ours against no_apf ( n=30 seeds for PPO, n=20 for SAC and TD3; error bars denote 95% confidence intervals).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Setting
Environment
PandaReach-v3 (dense reward), 100 steps per episode
Action
[−1,1]3 , scaled to a maximum step of 0.05m
Success threshold
0.03m ; discount factor γ=0.99
Pillars
N=2 , height 0.21m , nominal centers (−0.07,0.045) and (−0.19,0.015)m ; random walk along x with a step drawn uniformly from [−5,5]mm , reflected at ±15mm from the nominal center; reset to the nominal centers at every episode
Training radius
ρ=0.03m
Goal sampling
x∈[−0.38,−0.30] , y∈[0.03,0.15] , z∈[0.08,0.13]m , rejected if within ρ+0.03m of a nominal pillar center
Appendix
Table B.1: Environment, solver, and planner settings.
dsafe
λs
Clearance
Path
Safe success (%)
Collision (%)
Seeds
(mm)
(mm)
(m)
ρ=0.03
ρ=0.055
ρ=0.055
≥50%
40
1.5
25.3
0.522
91.6
53.2
42.2
9/10
60
1.5
36.8
0.528
93.2
81.8
12.6
10/10
80 (default)
1.5
46.5
0.581
95.2
89.8
5.8
10/10
100
1.5
56.5
0.600
92.0
84.2
9.2
9/10
80
0.75
25.0
0.515
91.0
51.8
42.8
7/10
Appendix
Table D.1: Sensitivity of ours to dsafe and λs ( n=10 seeds per row). Clearance and path length are measured at the training radius ρ=0.03m ; the safe success rate and the collision rate are percentages. The last column counts the seeds whose safe success rate at ρ=0.055m is at least 50% .
Robust and reactive collision avoidance remains a central challenge for deploying Reinforcement Learning (RL) policies on real robotic manipulators, particularly when transferring behaviours learned in simulation to the physical world. This paper presents an RL framework that integrates the Euclidean Distance Field (EDF) directly into the observation space, providing the agent with continuous geometric cues that support stable and responsive avoidance behaviours. By leveraging the rich spatial information encoded in the EDF, the method enables a streamlined reward formulation that achieves effective obstacle avoidance both with and without Artificial Potential Field guidance. The proposed approach is validated on a 7-DoF redundant manipulator, demonstrating zero-shot transfer from simulation to real hardware. Despite being trained exclusively with randomised static objects, the deployed policy generalizes to previously unseen real environments and exhibits robust online avoidance of dynamic obstacles. These results highlight the potential of EDF-based representations to mitigate the sim-to-real gap and support reliable and real-time collision-aware behaviour in complex and changing environments.
Taiga Ikeshima, Sheila Sutjipto, Lan Wu +5
Tokyo University of Science, Japan. · the University of Technology Sydney, Australia. · Universidad Carlos III de Madrid, Spain. +1
Emergency collision avoidance under extreme driving conditions demands safety-critical control that accounts for both obstacle proximity and vehicle dynamic stability over a future time horizon, yet existing methods often rely on instantaneous or local safety evaluations. This paper proposes a safe reinforcement learning framework guided by a Hamilton-Jacobi (HJ) reachability based motion safety set that provides forward-looking safety supervision for constrained policy optimization. Specifically, a unified signed safety function is formulated by combining geometric collision margins and chassis stability limits, and is then extended through reachability analysis into a finite-horizon motion safety set that characterizes whether safety can be maintained under future vehicle state evolution. To enable practical computation, the motion safety set is approximated from offline extreme driving data, mitigating the computational burden of grid-based HJ solvers. The learned motion safety set is then embedded as a continuous safety cost into a constrained Markov decision process, and a PID-Lagrangian policy optimization scheme is employed to adaptively regulate the Lagrange multiplier for safety constraint enforcement. Simulation and real-vehicle experiments on low-adhesion obstacle-avoidance scenarios demonstrate that the proposed method achieves higher goal-reaching rates, produces smoother avoidance maneuvers, and maintains larger unified safety margins than baseline methods.
Yuhong Jiang, Shiyue Zhao, Junzhi Zhang +4
School of Vehicle and Mobility, Tsinghua University, Beijing, China · State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University, Beijing, China
Policies for robotic manipulation are produced by training on large teleoperated datasets. These datasets typically consist of free-space trajectories, making them difficult to transfer to test-time environments with obstacles. Previous methods for closing this gap have largely fallen into two groups. Dataset augmentation addresses it at training time but needs obstacle geometry in advance, whereas steering an existing checkpoint at inference time avoids that requirement but is limited in flexibility. Our method draws from both areas without inheriting either drawback. DetAug applies an obstacle-blind augmentation scheme to the transit phases of a free-space dataset, leaving object interactions untouched, and records the augmentation parameters as an explicit conditioning label. At inference it samples a batch of labels and executes the trajectory with the lowest collision cost. On the SafeLIBERO benchmark DetAug achieves a collision-free success rate more than 20pp above the next best method, and selecting over the label space outperforms guidance on the same policy by 26pp. On real hardware, inference-time steering methods collapse on tasks requiring large detours, while DetAug matches or exceeds an obstacle-conditioned baseline without ever seeing obstacles in training.
Reece O'Mahoney, Moritz Zoellner, Ioannis Havoutis
Oxford Robotics Institute, University of Oxford, Oxford, UK · Purdue University, West Lafayette, IN, USA