Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation

Michal Pliska; Matouš Vrba; Ondřej Víta; Martin Jiroušek; Viktor Walter; Martin Saska

cs.ROSep 30, 2026

Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation

Authors: Michal Pliska, Matouš Vrba, Ondřej Víta, Martin Jiroušek, Viktor Walter, Martin Saska

Organizations: Multi-robot Systems Group, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic

Abstract

Agile multi-UAV flight requires accurate and low-latency onboard estimation of the kinematic states of neighboring UAVs for collision avoidance, motion coordination, etc. Most vision-based approaches rely on position-only measurements, inferring velocity and acceleration indirectly from displacement. We show that this introduces a fixed structural delay in the estimation of higher-order states, which limits the achievable agility. To address this, we propose to integrate tilt measurements, provided by a state-of-the-art visual detector, which inform about the thrust direction of co-planar multirotor UAVs. We benchmark four position-only and five pose-aware estimators, including a novel formulation of a linear thrust-constraining Kalman filter, on two real-world and one high-fidelity photorealistic simulated dataset over different levels of agility (3-21 m/s^2). In our setup, pose-aware estimation consistently reduces the average velocity and acceleration estimation errors by 40% and 57% across the three datasets with the proposed KF formulation outperforming the other estimators. Position-only filters exhibit a constant ~300 ms delay in acceleration step response independent of agility, whereas the tilt-constrained estimators operate near the physical response limit given by the camera frame-rate by observing the change in thrust direction before the displacement accumulates. In a closed-loop leader-follower simulated experiment with NMPC control, position-only estimation of the leader's state fails to facilitate stable hovering of the follower, while the proposed estimator enables tracking of lateral maneuvers exceeding 2g of acceleration.

Figures & tables

Fig. 1 : Onboard view from a follower UAV during closed-loop tracking at accelerations above $2\text{\,}\mathrm{g}$ . Red boxes show raw YOLOv5-6D detections. The first frame ( $t_{1}$ ) is the current image, the remaining frames ( $t_{2}$ – $t_{6}$ ) are a time-lapse during an aggressive lateral maneuver. The times $t_{1}$ – $t_{6}$ correspond to the markers in Fig. 5 .

Fig. 2 : Relation of the world frame

\mathcal{W}

and the uav ’s body frame

\mathcal{B}

through the position vector

\bm{p}

and orientation matrix

\mathbf{R}

. Basis vectors of these frames and the relevant acceleration components of the total uav acceleration

\bm{a}=\bm{a}_{T}+\bm{a}_{d}+\bm{a}_{g}

are also denoted.

Dataset	Unreal (ours)	Phantom 4 [ 10 ]	Mavic 2 [ 10 ]
Number of sequences	140	68	61
Total duration	$1333\text{\,}\mathrm{s}$	$667\text{\,}\mathrm{s}$	$668\text{\,}\mathrm{s}$
Number of frames	33460	16734	16755
Sampling period	$40\text{\,}\mathrm{ms}$	$40\text{\,}\mathrm{ms}$	$40\text{\,}\mathrm{ms}$
Camera motion	Flying	Static	Static
Camera range ∗	$1.58.0\text{\,}\mathrm{m}$	$1.95.1\text{\,}\mathrm{m}$	$1.95.1\text{\,}\mathrm{m}$

TABLE I : Comparison of the benchmarking datasets.

	Unreal (simulated)			Phantom 4 (real-world)			Mavic 2 (real-world)
Estimator	Pos. ( $\mathrm{m}$ )	Vel. ( $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ )	Acc. ( $\mathrm{m}\text{\,}{\mathrm{s}}^{-2}$ )	Pos. ( $\mathrm{m}$ )	Vel. ( $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ )	Acc. ( $\mathrm{m}\text{\,}{\mathrm{s}}^{-2}$ )	Pos. ( $\mathrm{m}$ )	Vel. ( $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ )	Acc. ( $\mathrm{m}\text{\,}{\mathrm{s}}^{-2}$ )
Position-only
CV-KF	0.100	2.300	–	0.131	0.755	–	0.070	0.793	–
CV-KF+BDC	0.112	0.739	–	0.131	0.415	–	0.071	0.426	–
CA-KF	0.104	0.791	4.899	0.131	0.487	2.337	0.093	0.609	2.440
CA-KF+BDC	0.099	0.787	4.989	0.131	0.415	2.039	0.078	0.488	2.102

TABLE II : men values for position, velocity, and acceleration across datasets and estimators.

Fig. 3 : Velocity and acceleration men for the different agility levels in the Unreal dataset. cv - kf is omitted due to a significantly higher error at all levels (see Table II ). Dots denote mean over 10 trajectories, lines indicate standard deviation.

Agility group	Low			Mid		High
Reference max. ( $\mathrm{m}\text{\,}{\mathrm{s}}^{-2}$ )	3.0	6.0	9.0	12.0	15.0	18.0	21.0
Flown acc. Q2 † ( $\mathrm{m}\text{\,}{\mathrm{s}}^{-2}$ )	1.6	3.0	4.0	5.7	7.0	7.9	10.0
Flown acc. p98 † ( $\mathrm{m}\text{\,}{\mathrm{s}}^{-2}$ )	2.8	5.7	7.9	11.1	14.0	15.9	19.1
Flown tilt p98 † ( $\mathrm{\SIUnitSymbolDegree}$ )	15	32	45	62	71	86	95
Num. of sequences	20	20	20	20	20	20	20
† Q2, p98 denote the $50\text{\,}\mathrm{\%}$ and $98\text{\,}\mathrm{\%}$ percentiles.

TABLE III : Agility levels in the Unreal (simulated) dataset.

	Low (3–9)		Mid (12–15)		High (18–21)
Estimator	$xy$	$z$	$xy$	$z$	$xy$	$z$
Velocity men ( $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ )
CA-KF+BDC	0.376	0.248	0.393	0.306	0.553	0.404
Pose-KF [ 10 ]	0.194	0.220	0.219	0.332	0.312	0.444
Z-KF+BDC	0.136	0.207	0.192	0.260	0.259	0.330
Relative improvement of Z-KF+BDC over CA-KF+BDC

TABLE IV : Per-axis men values on the Unreal dataset across the different agility groups.

Fig. 4 : Acceleration step response at

9\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-2}

magnitude. Shaded regions indicate variance across five runs. Crosses represent camera frames.

$a_{\mathrm{ref}}$	$\tau_{\text{gt}}$		$\tau_{\text{CA-KF+BDC}}$		$\tau_{\text{Z-KF+BDC}}$
( $\mathrm{m}\text{\,}{\mathrm{s}}^{-2}$ )	$\mathrm{ms}$	frames	$\mathrm{ms}$	frames	$\mathrm{ms}$	frames
3	152.0	3.8	456.4	11.4	156.0	3.9
6	159.2	4.0	482.9	12.1	208.4	5.2
9	164.4	4.1	477.6	11.9	247.3	6.2
12	172.1	4.3	474.8	11.9	292.1	7.3
15	180.6	4.5	476.1	11.9	327.0	8.2

TABLE V : Step response delay over reference magnitude.

Fig. 5 : Closed-loop leader-follower tracking with zlkf and NMPC. Note that with a position-only estimator, the follower is not even capable of stable hovering (not shown). Markers $t_{1}$ – $t_{6}$ reference Fig. 1 . (a) Position. (b) Velocity. (c) Acceleration.

Explore similar work

Image-Domain Tilt Constrained Distributed Fusion for Maneuvering UAV Tracking with Multi-Camera Electro-Optical Observations
Jul 1, 2026Minxing Sun, Yao MaoUnmanned Aerial Vehicle Detection Inertial Measurement Units
Institute of Optics and Electronics, Chinese Academy of Sciences, Chengdu, China · Institute for Infocomm Research (I2R), Agency for Science, Technology and Research, Singapore · Shenzhen Astralldynamics Technology
Robust Active-Perception Control for Global-State-Free Aerial-Ground Cooperation
Sep 22, 2026Mingxuan Zhang, Jiajun Yu, Baozhe Zhang +5Cooperative Air Combat Viewpoint-Dependent Active Perception
State Key Laboratory of Industrial Control Technology, Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, China · Huzhou Institute of Zhejiang University and Huzhou Key Laboratory of Autonomous Systems, Huzhou 313000, China · The Chinese University of Hong Kong, Shenzhen 518172, China
Motor Angular Speed Preintegration for Multirotor UAV State Estimation
Jun 18, 2026Matěj Petrlík, Filip Novák, Robert Pěnička +1Inertial Navigation Systems State Estimation
Department of Cybernetics, Faculty of Electrical Engineering, Czech Technical University in Prague, 166 36, Prague 6, Czech Republic

Jul 1, 2026, eess.IVJ/K move · Enter open · S save

Image-Domain Tilt Constrained Distributed Fusion for Maneuvering UAV Tracking with Multi-Camera Electro-Optical Observations

Minxing Sun, Yao Mao

Institute of Optics and Electronics, Chinese Academy of Sciences, Chengdu, China · Institute for Infocomm Research (I2R), Agency for Science, Technology and Research, Singapore · Shenzhen Astralldynamics Technology

Unmanned Aerial Vehicle Detection Inertial Measurement Units

Open paper AskPDF arXiv

Short-horizon prediction is essential for electro-optical UAV tracking, especially when the target is small, maneuvering, or intermittently observed. Image center, line-of-sight, and range measurements provide direct constraints on target position, but their constraints on acceleration are weak. As a result, prediction can lag during aggressive maneuvers. This paper proposes an image-domain tilt constrained distributed fusion method for maneuvering UAV tracking. The method uses the apparent roll and pitch of a rotorcraft target in the image as low-level maneuver cues. A weak-prior auto-labeling pipeline first generates oriented bounding box and image-domain tilt labels from synchronized video, gimbal IMU, and UAV IMU data. A YOLO-OBB detector is then trained to provide online target position and tilt measurements. The front-end Python implementation is publicly available at github.com/ShineMinxing/PythonYOLO. In the fusion stage, the UAV state is modeled by position, velocity, and acceleration. Image-domain roll and pitch are introduced as acceleration-related pseudo-observations. For distributed tracking, one mobile gimbal camera and two fixed ground cameras are fused asynchronously. Camera attitude error states are augmented into the filter to absorb extrinsic drift and cross-camera systematic inconsistency. A Mahalanobis gate with time-since-last-valid covariance widening is used to reject false detections and handle dropouts. In simulation, adding roll/pitch observations reduces the prediction RMSE from 1.991 m to 0.821 m and decreases the cumulative prediction error by 60.75%. In real distributed experiments, a self-consistency evaluation shows an 18.10% reduction in cumulative prediction error. The results show that image-domain tilt can provide useful acceleration constraints for robust short-horizon UAV prediction.

Abstract

Explore similar work

Image-Domain Tilt Constrained Distributed Fusion for Maneuvering UAV Tracking with Multi-Camera Electro-Optical Observations

Robust Active-Perception Control for Global-State-Free Aerial-Ground Cooperation

Motor Angular Speed Preintegration for Multirotor UAV State Estimation