Robust Tightly-Coupled Filter-Based Monocular Visual-Inertial State Estimation and Graph-Based Evaluation for Autonomous Drone Racing
Authors: Maulana Bisyir Azhari, Donghun Han, Sung Jun Park, David Hyunchul Shim
Organizations: Unmanned Systems Research Group (USRG), School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, South Korea.
Autonomous drone racing (ADR) demands state estimation that is simultaneously computationally efficient and resilient to the perceptual degradation experienced during extreme velocity and maneuvers. Traditional frameworks typically rely on conventional visual-inertial pipelines with loosely-coupled gate-based Perspective-n-Points (PnP) corrections that suffer from a rigid requirement for four visible features and information loss in intermediate steps. Furthermore, the absence of GNSS and Motion Capture systems in uninstrumented, competitive racing environments makes the objective evaluation of such systems remarkably difficult. To address these limitations, we propose ADR-VINS, a robust, monocular visual-inertial state estimation framework based on an Error-State Kalman Filter (ESKF) tailored for autonomous drone racing. Our approach integrates direct pixel reprojection errors from gate corners features as innovation terms within the filter. By bypassing intermediate PnP solvers, ADR-VINS maintains valid state updates with as few as two visible corners and utilizes robust reweighting instead of RANSAC-based schemes to handle outliers, enhancing computational efficiency. Furthermore, we introduce ADR-FGO, an offline Factor-Graph Optimization framework to generate high-fidelity reference trajectories that facilitate post-flight performance evaluation and analysis on uninstrumented, GNSS-denied environments. The proposed system is validated using TII-RATM dataset, where ADR-VINS achieves an average RMS translation error of 0.143 m, while ADR-FGO yields 0.060 m as a smoothing-based reference. Finally, ADR-VINS was successfully deployed in the A2RL Drone Championship Season 2, maintaining stable and robust estimation despite noisy detections during high-agility flight at top speeds of 20.9 m/s. We further utilize ADR-FGO for post-flight evaluation in uninstrumented racing environments.
Figures & tables
Fig. 1: Left: The autonomous drone racing platform used in the A2RL Drone Championship Season 2, featuring a low-cost monocular RGB camera and IMU sensors. Right: Autonomous racing flight at high speeds over 20 m/s featuring the proposed ADR-VINS for its real-time state estimation.
Ref
Validation
Sensors
Gate/Corner Detection
State Estimation Pipeline
Top Speed
Real-World
Dataset
Camera
IMU
Additional
Gate
Corner
Type
[ 2 ]
AIRR ∗
-
Stereo
Yes
1D LiDAR
-
U-Net
1-stage †
ROVIO + Drift EKF
8 m/s
[ 3 ]
AIRR ∗
-
Mono
Yes
1D LiDAR
GateNet
Snake-Gate
2-stages
RANSAC + PnP + MHE
9.2 m/s
[ 4 ]
Self-hosted
-
Stereo
Yes
-
-
U-Net
1-stage †
T265-VIO + PnP + Drift KF
22 m/s
[ 5 ]
Self-hosted
-
Stereo
Yes
-
YoloV8
U-Net
2-stages
T265-VIO + PnP + Drift KF + EKF
21.1 m/s
[ 6 ]
A2RL S1 ∗
-
Mono
Yes
-
-
U-Net
1-stage †
VINS-Mono/OpenVINS + PnP + Drift KF
12.5 m/s
TABLE I: Comparison of Gate-Based State-Estimation Methods for Autonomous Drone Racing.
Fig. 2: Overview of ADR-VINS. (a) The framework consists of a visual measurement pipeline (including detection, reordering, and association) and a tightly-coupled Error-State Kalman Filter (ESKF) that fuses raw IMU data with direct pixel reprojection errors to provide robust, high-rate state estimation. (b) Illustration of the corner detections (yellow dots), reprojections (red dots), and the corresponding reprojection errors (red lines).
Fig. 3: The proposed factor graph structure in ADR-FGO. The graph integrates IMU pre-integration, gate corner detections, pose prior factors from ADR-VINS, and camera extrinsics priors. Keyframes are maintained even during visual outages to distribute drift and IMU biases across these visual-less sections to provide better interpolation.
Seq
ONLINE METHODS
OFFLINE METHODS
OpenVINS
PnP-Pose
PnP+EKF
ADR-VINS
MAPLAB
ADR-FGO
et
er
ev
et
er
ev
et
er
ev
et
er
ev
et
er
ev
et
er
ev
Piloted
05P
.296
1.77
.293
.466
4.05
-
.282
1.59
.388
.130
1.73
.202
.194
1.60
.153
.052
1.16
.090
06P
.943
2.21
.449
.488
4.17
-
.350
2.64
.505
.134
1.69
.218
.159
1.35
.132
.061
1.47
.100
11P
.369
3.12
.345
.398
3.39
-
.242
1.76
.371
.132
1.85
.199
.249
2.20
.216
.054
1.25
.110
12P
.204
2.39
.194
.298
3.87
-
.245
1.62
.355
.169
2.02
.221
.297
1.67
.208
.049
1.39
.090
TABLE II: Results on the TII-RATM Dataset.
Fig. 4: Qualitative comparison of ADR-VINS against baseline online methods across various tracks on the TII-RATM dataset autonomous sequences, including an ellipse (Flight 06A), a lemniscate (Flight 12A), and a complex 3D racing track (Flight 18A).
Fig. 5: Comparison of ADR-VINS with # 2, # 4, and # 6 minimum corners visibility requirements for Flight 18A. Requiring only two corners enables earlier and more frequent updates, significantly reducing drift. ADR-FGO further minimizes this error by globally refining ADR-VINS, even in sections where gates are not visible. G1-G7 is the gate ordering.
w/o vl KF
w/o ext factor
Full
Seq.
et
er
ev
et
er
ev
et
er
ev
Avg @P
.056
1.51
.111
.067
1.88
.115
.054
1.32
.098
Avg @A
.064
2.37
.177
.070
2.15
.179
.064
2.13
.165
TABLE III: ADR-FGO Ablation Study.
Fig. 6: Top: Visualization of reprojection errors for ADR-VINS and ADR-FGO compared to raw detections on flight 18A. Bottom: Statistics of the detection and reprojection errors on TII-RATM dataset.
Fig. 7: Top: 3D visualization of ADR-VINS against GT (ADR-FGO) of our fastest flight in A2RL S2 track. Middle: Velocity and acceleration profile. Bottom: Error analysis.
PnP-Pose
PnP+EKF
ADR-VINS
et
er
ev
et
er
ev
et
er
ev
.797
5.97
-
.367
2.31
.717
.152
1.25
.413
TABLE IV: Results on A2RL Season 2 Track.
Fig. 8: Top: Visualization of the corner detections and the reprojections of ADR-VINS and ADR-FGO estimates. Bottom: Statistics of the detection and reprojection errors on our fastest flight in A2RL S2.
Fig. 9: Robustness ablation study visualization. Performance comparison of the proposed Huber-based reweighting, a χ2 (chi-squared) rejection test, and naive visual measurements fusion. The Huber-based method maintains stability against noisy detections where naive and χ2 test variants exhibit significant drift or failure.
Visual Measurements
ESKF
component:
detection
reorder.+assoc.
propagation
update
time [ms]:
8.165
0.192
0.011
0.242
TABLE V: Timing of ADR-VINS.
Fig. 11: Robustness against added detection noise in TII-RATM Dataset.
Autonomous drone racing demands robust real-time localization under extreme conditions: high-speed flight, aggressive maneuvers, and payload-constrained platforms that often rely on a single camera for perception. Existing visual SLAM systems, while effective in general scenarios, struggle with motion blur and feature instability inherent to racing dynamics, and do not exploit the structured nature of racing environments. In this work, we present a dual pose-graph architecture that fuses odometry with semantic detections for robust localization. A temporary graph accumulates multiple gate observations between keyframes and optimizes them into a single refined constraint per landmark, which is then promoted to a persistent main graph. This design preserves the information richness of frequent detections while preventing graph growth from degrading real-time performance. The system is designed to be sensor-agnostic, although in this work we validate it using monocular visual-inertial odometry and visual gate detections. Experimental evaluation on the TII-RATM dataset shows a 56% to 74% reduction in ATE compared to standalone VIO, while an ablation study confirms that the dual-graph architecture achieves 10% to 12% higher accuracy than a single-graph baseline at identical computational cost. Deployment in the A2RL competition demonstrated that the system performs real-time onboard localization during flight, reducing the drift of the odometry baseline by up to 4.2 m per lap.
David Perez-Saura, Miguel Fernandez-Cortizas, Alvaro J. Gaona +1
Computer Vision & Aerial Robotics Group, Universidad Polit´ecnica de Madrid, Madrid, Spain · Automation and Robotics Group (ARG-SnT), University of Luxembourg – SnT, Luxembourg
Nanodrones require accurate, real-time state estimation under severe sensing and computational constraints. We present TinyCVIO, a visual-inertial odometry system that co-designs miniature sensing, visual processing, and estimation for a commodity dual-core microcontroller with 520 kB SRAM. Lightweight LED constellations provide known geometry without surveyed positions or yaw angles, assuming placement on a common level plane. A streaming visual frontend tracks LED observations from a millimeter-scale camera at 29.2 FPS, while a rigid-board measurement model retains inter-LED constraints and streaming QR bounds estimation workspace for a fixed filter-state size. Across 19 hand-held hardware-in-the-loop datasets, the rigid-board model reduces mean absolute trajectory error by 27% relative to planar points. The complete system runs onboard a Crazyflie across nine flights at three speeds, achieving 3.5-3.7 cm mean absolute trajectory error and 0.50-0.60% relative pose error over 10 m segments, with mean estimate latency of 15.7-16.3 ms.
Derin Ozturk, Kaan Akan, Irwin Wang +1
Department of ECE, Cornell University, Ithaca, NY, USA
The robustness of event cameras to high dynamic range and motion blur holds the potential to improve visual odometry systems in challenging environments. Although their high temporal resolution does not require synchronous processing, most event-based odometry methods still run at fixed rates, which simplifies system design but restricts latency and throughput. In this work, we present AERO-VIS, a stereo event-inertial SLAM system with an integrated, data-driven, robust, and performance-optimized keypoint detector. By decoupling event preprocessing from state estimation, our architecture operates asynchronously at the maximum rate supported by available hardware, dynamically adapting to downstream runtime demands and ensuring low-latency, real-time performance. When deploying AERO-VIS on a UAV, we achieve unprecedented accuracy in onboard event-based SLAM. These unique characteristics enable us to present the first purely event-based inertial SLAM system that demonstrates closed-loop UAV control and large-scale state estimation while relying solely on onboard compute. Videos of the experiments, source code, and additional results are available at https://ethz-mrl.github.io/AERO-VIS/.
Yannick Burkhardt, Sebastián Barbas Laina, Simon Boche +2
Mobile Robotics Lab, ETH Zürich · Technical University of Munich · Munich Center for Machine Lerning (MCML) +1