Formation-Aware Adaptive Conformalized Perception for Safe Leader-Follower Multi-Robot Systems
Authors: Richie R. Suganda, Bin Hu
Organizations: Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77004, USA. · Department of Engineering Technology, University of Houston, Houston, TX 77004, USA.
This paper considers the perception safety problem in distributed vision-based leader-follower formations, where each robot uses onboard perception to estimate relative states, track desired setpoints, and keep the leader within its camera field of view (FOV). Safety is challenging due to heteroscedastic perception errors and the coupling between formation maneuvers and visibility constraints. We propose a distributed, formation-aware adaptive conformal prediction method based on Risk-Aware Mondrian CP to produce formation-conditioned uncertainty quantiles. The resulting bounds tighten in high-risk configurations (near FOV limits) and relax in safer regions. We integrate these bounds into a Formation-Aware Conformal CBF-QP with a smooth margin to enforce visibility while maintaining feasibility and tracking performance. Gazebo simulations show improved formation success rates and tracking accuracy over non-adaptive (global) CP baselines that ignore formation-dependent visibility risk, while preserving finite-sample probabilistic safety guarantees. The experimental videos are available on the project website\footnote{Project Website: https://nail-uh.github.io/iros2026.github.io/}.
Figures & tables
Fig. 1: Formation-Aware Adaptive Conformalized Leader–Follower System. Under heteroscedastic, formation-dependent perception errors, the proposed risk-aware conformal bounds tighten in low-risk configurations to preserve tracking performance and expand near FOV limits to enforce probabilistic safety.
Fig. 2: Risk-Aware Adaptive Conformal Control for Safe Leader–Follower Formation Architecture . Onboard sensing feeds a risk-aware Mondrian conformal predictor that produces state-dependent uncertainty bounds, which parameterize the formation-aware conformal CBF-QP safety filter to enable safe tracking under perception uncertainty.
Fig. 3: Empirical nonconformity-score distributions across formation-dependent risk regions. The scores are strongly heteroscedastic: the high-risk near-boundary set B1 exhibits substantially larger variance and heavier tails than the low-risk interior set B3 . This pronounced risk-dependent error structure motivates a formation-aware (risk-aware) conformal calibration that assigns region-specific quantiles, rather than a single global bound.
Fig. 4: Risk-aware Mondrian conformal quantiles projected over the camera FOV: uncertainty increases near the safety boundaries (red) and decreases toward the interior (green).
Fig. 5: Linear interpolation of the risk-aware conformal quantiles across Mondrian risk partitions, yielding a continuous margin q~i(h) as in ( 10 ).
Fig. 6: Gazebo race-track environment and benchmark maneuver illustrating the three formation-dependent perception risk regimes: low-risk tracking near the FOV center (green), medium-risk during sustained turning between center and boundary (orange), and high-risk operation near the FOV boundary (red).
Fig. 7: Performance comparison over 100 trials. The proposed Risk-Aware Method achieves the highest success rate and it also yields the best distance and bearing tracking statistics among the safe methods, while using substantially smaller conformal margins ( q~ ) than the high-risk global bound, reducing conservatism via formation-dependent adaptation. ∗ Nominal and low-risk tracking errors are computed only on successful runs (survivorship bias).
Fig. 8: Average transient performance over 100 trials. The Risk-Aware Method achieves the best transient tracking, particularly in bearing during the high-risk segment near the FOV boundary. The bottom plot shows the formation-dependent conformal margin q~ shrinking in low-risk regions and expanding in high-risk regions, preserving safety while avoiding unnecessary conservatism. Averages for the nominal and global CP baselines are computed over their successful runs only.
This letter proposes a distributed 3D leader-follower formation (3D-LFF) control framework for multi-UAV systems that achieves formation tracking while enforcing perception safety constraints. Maintaining safe, vision-based 3D-LFF is challenging because onboard cameras impose strict Field-of-View (FOV) limitations, and demanding formation commands can drive the leader outside the follower's camera frustum, resulting in loss of visibility. To address this issue, we develop a perception-aware safe control architecture that guarantees visibility by construction. First, we derive a relative kinematic model in a line-of-sight coordinate representation and design a distributed 3D-LFF tracking controller using only locally available relative states. Next, we embed the nominal formation controller within a Control Barrier Function-based Quadratic Program (CBF-QP) safety filter that minimally modifies the commanded velocities to maintain the leader inside the follower's camera frustum while preserving formation tracking whenever feasible. Gazebo simulations and Crazyflie hardware experiments validate the proposed approach, demonstrating accurate formation tracking and effective FOV enforcement, including scenarios in which the nominal desired formation conflicts with visibility constraints.
Immanuel R. Santjoko, Richie R. Suganda, Miao Pan +1
Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77004, USA · Department of Engineering Technology, University of Houston, Houston, TX 77004, USA
Classical leader-follower formation control suffers from single points of failure and error propagation, and relies on absolute localization sensors that are ill-suited for GPS-denied environments. We present a learned, vision-only estimator that maps each robot's monocular image, together with messages exchanged over a communication graph, directly to its 6-DoF relative pose. Its key ingredient is the implicit virtual leader (IVL): a non-physical reference frame at the team centroid, implicitly learned inside a Transformer-based graph neural network, so that estimation has no privileged node and needs no absolute localization. The estimator additionally reports well-calibrated aleatoric (heteroscedastic GNLL) uncertainty alongside epistemic (MC~Dropout) uncertainty, compared systematically across simulation and real-world test sets. Trained only in simulation, the estimator generalizes to unseen scenes, to larger unseen team sizes, and to an external real-world benchmark. It exhibits no single point of failure: removing any one robot costs at most 1.24× the median removal, and removing 71% of the communication links costs 1.77× in position error without retraining. Trained on real-robot data from a single platform, it transfers without modification to a heterogeneous team, estimating relative pose to 0.22,m and 1.6∘ on physical robots, where it drives closed-loop formation control.
Cooperation in multi-UAV systems requires reliable relative perception so that follower vehicles can maintain formation and continue their mission safely even when absolute positioning sensors degrade or fail. This paper presents a vision-based cooperative formation framework running on a follower UAV that uses a front-facing RGB-D camera to detect, track, and localize a leader UAV in real-time. A lightweight YOLO-based detector is trained on a dedicated drone dataset and deployed onboard to predict leader bounding boxes, which are then fused with depth information via a pinhole camera model to estimate the leader's relative pose. These estimates provide a leader-follower position controller and can also be used as a backup when GPS or external localization is unavailable. This framework is implemented as a set of ROS nodes and evaluated in a physics-based multi-UAV simulation built on XTDrone, with sensor noise and communication dropouts. We evaluate detection accuracy, runtime, and formation-keeping error under nominal conditions and under simulated failures of the positioning sensors. The results show that the proposed framework maintains stable leader-follower formations with reasonable computational cost and provides a practical basis for extending vision-based cooperative formation control to real-world multi-UAV systems.
AI, IoT and Robotics Lab (AIR Lab), UAVs Group, Autonomous Robotics Systems Limited, Hyderabad, India · Department of Microelectronics and VLSI Design, University of Hyderabad, Hyderabad, India · Department of Internet of Things, Ideabytes Software India Private Limited, Hyderabad, India +5