Distilling Privileged Control Barrier Functions into RGB-Only Safety Filters for Dynamic Visual Navigation

Seungyeon Yoo; Gawon Lee; Seungwoo Jung; Inkyu Jang; H. Jin Kim

cs.ROSep 29, 2026

Distilling Privileged Control Barrier Functions into RGB-Only Safety Filters for Dynamic Visual Navigation

Authors: Seungyeon Yoo, Gawon Lee, Seungwoo Jung, Inkyu Jang, H. Jin Kim

Organizations: Department of Aerospace Engineering, Seoul National University, Seoul, Republic of Korea

Abstract

RGB-only end-to-end visual navigation policies remain vulnerable to collisions in real-world dynamic environments, motivating a dedicated safety layer. Existing visual Control Barrier Function (CBF) approaches seek to provide safety from RGB observations, but often rely on real-time rendering or explicit scene reconstruction and are primarily designed for static scenes, limiting their practicality for onboard deployment. We propose a teacher-student visual distillation framework that transfers the safety behavior of a privileged CBF teacher to an RGB-only student filter for dynamic environments. The student maps a short RGB history, robot velocity, and a nominal control action directly to a safe action, while the teacher uses ground-truth robot and obstacle states in a real-to-sim dynamic Gaussian Splatting environment. To reduce the teacher-student information gap, the teacher constructs safety constraints only from obstacles observable within the student's RGB history. It also accounts for obstacle-velocity uncertainty to improve robustness to motion variations, while action augmentation exposes the student to diverse safe and unsafe nominal actions to better capture the safety boundary. At deployment, the student requires only RGB observations and robot velocity, without explicit 3D reconstruction or online rendering. Experiments show that the proposed method outperforms visual CBF baselines and improves the safety of RGB-based navigation policies under dynamic obstacle motion. Project page: https://syeon-yoo.github.io/distill-cbf-site/.

Figures & tables

Fig. 1: Real-world navigation example of the proposed RGB-only safety filter. We distill a privileged state-based CBF teacher into an RGB-only student filter in a real-to-sim simulation with moving obstacles. In real-world dynamic environments, the proposed filter improves both collision avoidance and task success of nominal visual navigation policies, without requiring a 3D scene representation, robot localization, or obstacle state estimation at deployment.

Fig. 2: Training pipeline. We train the RGB-only student safety filter by distilling a privileged teacher safety filter. The teacher draws its privileged state from the simulation, while the student learns to reproduce the teacher’s safe action from the observations available at deployment alone.

		Outside					Lobby					Library
Nominal	Safety filter	SR $\uparrow$	CFR $\uparrow$	$\bar{m}\uparrow$	$T$	Failure cases $\downarrow$	SR $\uparrow$	CFR $\uparrow$	$\bar{m}\uparrow$	$T$	Failure cases $\downarrow$	SR $\uparrow$	CFR $\uparrow$	$\bar{m}\uparrow$	$T$	Failure cases $\downarrow$
		(%)	(%)	(m)	(s)	S/D/TO (%)	(%)	(%)	(m)	(s)	S/D/TO (%)	(%)	(%)	(m)	(s)	S/D/TO (%)
IL	None	63	64	0.26	10.2	17/19/1	65	65	0.30	8.4	4/31/0	56	56	0.19	9.6	25/19/0
	Teacher	88	98	0.37	12.6	0/2/10	85	87	0.41	10.4	0/13/2	89	93	0.34	12.2	0/7/4
	SAFER-Splat	72	81	0.30	11.0	6/13/9	71	72	0.33	8.8	0 /28/ 1	67	72	0.25	10.4	14/14/5
	NeRF-CBF	47	74	0.19	16.3	16/10/27	22	76	0.11	16.8	1/23/54	65	68	0.23	12.7	20/12/ 3

TABLE I: Simulation results. Each nominal–filter–scene combination is evaluated over 100 trials. Best values are in bold.

Fig. 3: Reproducing the teacher’s action-space filtering boundary. The top row shows the robot’s observations in simulation, and the bottom row shows the corresponding action-space boundaries. The color indicates the teacher CBF constraint margin, with the dashed gray curve denoting its zero-level boundary. The green curve denotes the student’s intervention boundary. The results show that the RGB-only student’s boundary closely matches the teacher’s boundary.

Safety filter	SR $\uparrow$	CFR $\uparrow$	Failure cases $\downarrow$
	(%)	(%)	S/D/TO (%)
None	48.9	53.0	25.4/21.6/4.1
Teacher	76.8	91.5	1.2/ 7.3 /14.7
Teacher (w/o margin )	74.6	87.4	0.6 /12.1/ 12.8
Ours	74.0	86.1	3.1/ 10.8 /12.1
Ours (w/o margin )	73.6	82.8	1.9 /15.3/ 9.2

TABLE II: Effect of the uncertainty-aware margin. (w/o margin): the teacher without the uncertainty-aware margin on the human’s velocity, and our filter distilled from its labels.

Mixed	Aug.	SR $\uparrow$	CFR $\uparrow$	Failure cases $\downarrow$
		(%)	(%)	S/D/TO (%)
Teacher		76.8	91.5	1.2/7.3/14.7
–	–	72.2	83.6	4.7/11.7/ 11.4
–	✓	69.7	81.6	4.7/13.8/11.9
✓	–	70.7	83.9	4.4/11.7/13.2
✓	✓	74.0	86.1	3.1 / 10.8 /12.1

TABLE III: Effect of dataset aggregation and augmentation. Mixed : The student is trained on the union of the datasets collected under all three nominal policies rather than its own nominal’s. Aug. : Action augmentation of the CBF-QP labels.

		Outside					Lobby					Library
Nominal	Safety filter	SR $\uparrow$	CFR $\uparrow$	$\bar{m}\uparrow$	$T$	Failure cases $\downarrow$	SR $\uparrow$	CFR $\uparrow$	$\bar{m}\uparrow$	$T$	Failure cases $\downarrow$	SR $\uparrow$	CFR $\uparrow$	$\bar{m}\uparrow$	$T$	Failure cases $\downarrow$
		(%)	(%)	(m)	(s)	S/D/TO (%)	(%)	(%)	(m)	(s)	S/D/TO (%)	(%)	(%)	(m)	(s)	S/D/TO (%)
IL	None	30	40	0.13	41.1	50/10/10	70	70	0.32	26.0	0/30/0	50	50	0.20	33.4	0/50/0
IL	Ours	50	80	0.07	36.8	20/0/30	90	90	0.41	28.1	0/10/0	80	80	0.34	29.0	0/20/0
RL	None	50	50	0.18	34.9	20/30/0	20	20	0.05	43.3	50/30/0	50	50	0.13	32.8	10/40/0
RL	Ours	80	80	0.30	22.8	10/10/0	60	80	0.24	33.9	0/20/20	70	70	0.33	30.0	0/30/0

TABLE IV: Real-world navigation results. Each nominal–filter–scene combination is evaluated over 10 trials.

Fig. 4: Safety under aggressive forward commands. The nominal controller continuously commands forward motion with zero angular velocity, while the RGB-only safety filter modifies the commands in response to obstacles. (a) Library: the filter stops the robot to yield to a moving human (left) and steers left to avoid a static obstacle (right). (b) Lobby: the filter steers right to avoid the wall (left) and stops to wait for a moving human to pass (right).

Explore similar work

Jun 10, 2026cs.RO

SAFER-Nav: Enhancing Safety for Visual Robot Navigation via Segmentation-Aware Fine-Tuning

Vision-based navigation models, particularly foundation models, generate viable trajectories from RGB observations alone. However, even state-of-the-art transformer- and diffusion-based policies struggle to generalize in unfamiliar deployment environments containing unseen obstacles or shifted conditions. The resulting trajectories often remain goal-directed but unsafe. Existing efforts improve safety through external trajectory correction or internal geometric priors, yet the resulting policies are not trained to explicitly represent obstacle boundaries or traversable free-space structure. To address this, we propose a navigation model that incorporates these structures directly into the policy via fine-tuning and is designed to be compatible with diverse RGB-based backbones. Across multiple robot platforms, indoor environments, and static and dynamic obstacle scenarios, our method reduces collision frequency relative to ViNT, NoMaD, and their CARE-augmented variants while maintaining goal-reaching performance.

Geonyeong Ko, Giung Lee, Changjoo Nam

Dept. of Electronic Engineering, Sogang University, Seoul, Korea · Dept. of Computer Science, Rice University, Houston, TX, USA · Vertical Labs, Co., Ltd., Seoul, Korea

Robot Navigation Obstacle Avoidance Geometric Priors Model Fine-Tuning

Ask PDF

Mar 6, 2026cs.RO

CN-CBF: Composite Neural Control Barrier Function for Robot Navigation in Dynamic Environments

Safe navigation of autonomous robots remains one of the core challenges in the field, especially in dynamic and uncertain environments. One prevalent approach is safety filtering based on control barrier functions (CBFs), which are easy to deploy but difficult to design. Motivated by the shortcomings of existing learning- and model-based methods, we propose a simple yet effective neural CBF design method for safe robot navigation in dynamic environments. We employ the idea of a composite CBF, where multiple neural CBFs are combined into a single CBF. Individual CBFs are trained using data generated offline via the Hamilton-Jacobi reachability framework to approximate the optimal safe set for single moving obstacles. Additionally, we use a residual neural architecture, ensuring that the estimated safe set does not intersect with the corresponding failure set. The method is extensively evaluated in simulation experiments for a ground robot and a quadrotor, comparing it against several baseline methods. The proposed method improves success rates by up to 18% over the strongest baseline, while maintaining comparable or lower path lengths and motion times. The method is also demonstrated in hardware experiments for both types of robots.

Bojan Derajić, Sebastian Bernhard, Wolfgang Hönig

Technical University of Berlin, Germany · AUMOVIO, Germany · Robotics Institute Germany (RIG), Germany

Control Barrier Functions Robot Navigation Hamilton-Jacobi Reachability Dynamic Environments

Ask PDF

Oct 16, 2025cs.RO

CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions

Reinforcement learning (RL), while powerful and expressive, can often prioritize performance at the expense of safety. Yet safety violations can lead to catastrophic outcomes in real-world deployments. Control Barrier Functions (CBFs) offer a principled method to enforce dynamic safety -- traditionally deployed online via safety filters. While the result is safe behavior, the fact that the RL policy does not have knowledge of the CBF can lead to conservative behaviors. This paper proposes CBF-RL, a framework for generating safe behaviors with RL by enforcing CBFs in training. CBF-RL has two key attributes: (1) minimally modifying a nominal RL policy to encode safety constraints via a CBF term, (2) and safety filtering of the policy rollouts in training. Theoretically, we prove that continuous-time safety filters can be deployed via closed-form expressions on discrete-time roll-outs. Practically, we demonstrate that CBF-RL internalizes the safety constraints in the learned policy -- both enforcing safer actions and biasing towards safer rewards -- enabling safe deployment without the need for an online safety filter. We validate our framework through ablation studies on navigation tasks and on the Unitree G1 humanoid robot, where CBF-RL enables safer exploration, faster convergence, and robust performance under uncertainty, enabling the humanoid robot to avoid obstacles and climb stairs safely in real-world settings without a runtime safety filter.

Lizhi Yang, Blake Werner, Massimiliano de Sa +1

Caltech MCE

Control Barrier Functions Safety Filters

Ask PDF