Reference-frame stabilization registers every frame of a hovering-drone video onto one reference frame, keeping object positions comparable. The registration pipelines it relies on are selected on feature-matching benchmarks in which only the camera moves. Over a busy intersection, vehicles moving together supply mutually consistent correspondences that can draw the robust homography estimate toward their motion: consensus capture, which inlier statistics cannot flag. A two-part benchmark measures it and the object masking meant to prevent it. A static protocol scores ten pipelines under exactly known homographies on 4K aerial frames, tuned on a site-disjoint split at a matched resolution and feature budget. A dynamic protocol scores them at an offline and a real-time budget on hovers selected from the video alone to span the amount and coherence of moving foreground, plus a stationary control. It measures background alignment, object position error and the share of a moving group's displacement an estimate follows, against a pseudo-ground-truth reference audited with manually annotated ground control points, which also score every candidate. On static scenes, tuned AKAZE, RootSIFT, SIFT and BRISK lead, retuning the matcher thresholds raises the mean average accuracy (mAA) by at most 0.0062, and masking costs less than 0.005. Unmasked ORB is captured on 28 of 304 real-time dynamic pairs and on 6 offline; its real-time captured estimates have a median corner error of 18.8 px yet typical inlier counts and ratios. Masking removes capture for every pipeline and leaves the leading pipelines unchanged. Code, data and results are released.
Figures & tables
Benchmark
Content
Reference geometry
Independent motion
HPatches ( Balntas et al., 2017a )
planar surfaces
manual homography
✗
Efe et al. (2021a)
planar surfaces
manual homography
✗
IMC-PT ( Jin et al., 2021a )
landmark photos
SfM poses
✗
HEB ( Barath et al., 2023a )
landmark photos
SfM planes
✗
AdelaideRMF ( Wong et al., 2011a )
multi-structure pairs
none (memberships)
✓
Ljubičić et al. (2021a)
river videos
none (static features)
✓
Table 1: Benchmarks for feature matching, homography estimation and stabilization, and the property this work adds. “Independent motion” asks whether any part of the scene moves between the two views independently of the camera.
Symbol
Meaning
Ω , H×W
pixel domain of a recording, and its size
Fk , Fref
frame k ; the reference frame
H
group of planar homographies, PGL(3,R)
H⋅x , \dehom
action of H on a Euclidean point; homogeneous projection
Hk→ref , Href→k
true transform current → reference, and its inverse
Hk→ref
a candidate pipeline’s estimate of Hk→ref
Table 2: Notation used throughout. Transform superscripts give the direction of the mapping, so Href→k=(Hk→ref)−1 . Matrices and vectors are bold and scalars italic, so the homography H is distinct from the frame height H ; frame dimensions are H×W and object box extents w×h , following the usual image and detection conventions.
Case
Cell
Reference
Moving [%]
Coherence
Stationarity
Objects
2022-10-06_D10_PM5_U1
high moving / coherent
1170
1.94
0.80
0.25
67
2022-10-06_D3_PM3_F2
high moving / coherent
259
1.43
0.72
0.17
40
2022-10-06_D9_AM3_S2
high moving / mixed
300
1.89
0.22
0.48
76
2022-10-06_D1_PM1_B2
high moving / mixed
240
1.48
0.22
0.50
70
2022-10-06_D7_PM1_P1
low moving / coherent
300
0.17
0.74
0.90
44
2022-10-06_D3_AM1_M1
low moving / coherent
237
0.18
0.59
0.79
18
Table 5: Part II: the cases, selected from 831 candidate reference frames in 370 hovers of 2022-10-06. Candidates are points every 900 frames inside each hover, clear of its first and last seconds, and a case’s reference frame is its selected point; a hover supplies at most one case. Each candidate is read on pairs of its reference frame with the frames 1, 2, 5, 11 and 22 frames later. Moving: the share of the frame covered by reference-frame objects displaced by more than 0.1 median object widths; Coherence: the share of that foreground moving with one of them (within 0.1 widths or 25% of its displacement, whichever is larger); Stationarity: the share of the found foreground that stayed in place; each averaged over the offsets. Cells split at the pool medians (moving 0.67%, coherence 0.42); coherence is undefined below 4 displaced objects, which left 72 candidates unclassified, and the control comes from the top decile of stationarity. Reference: the reference frame’s index in its recording; Objects: its detections. Every cell was filled.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Perturbation
Range
What it stands for
Translation
±10\text{,}\mathrm{%}$$ of W and of H
wind drift
Rotation
±15\text{,}\mathrm{\SIUnitSymbolDegree}$$ about the frame center
yaw
Scale
±5\text{,}\mathrm{%}$$
altitude change
Projective
±5\text{\times}{10}^{-5}$$ on both projective entries
pitch and roll
Brightness
factor in [0.75,1.25]
exposure drift, time of day, cloud shadow
Saturation
factor in [0.95,1.05] , in HSV
color rendition
Appendix
Table 1: Part I perturbation. A scalar range v means uniform on [−v,+v] for additive quantities and on [1−v,1+v] for multiplicative ones. The rotation sweep fixes the rotation at one magnitude per run, narrows translation to ±5\text{,}\mathrm{%} and scale to $\pm$2\text{\,}\mathrm{\%} without a projective term, and disables the photometric chain; the deployment-regime configuration narrows the geometric ranges as stated in Section 6.1 . The geometric ranges are those of the published campaign.
Parameter
Candidates
Reference estimator
Matcher
brute force, ratio test (LightGlue for its detectors; none for LoFTR)
brute force, ratio test
Ratio-test threshold
tuned ( Table 1 )
tuned ( Section 7.2 )
Inlier threshold
tuned ( Table 1 )
tuned ( Section 7.2 )
Robust estimator
MAGSAC++ (OpenCV flag 38), projective model
same
Iteration cap
5000
20000
Confidence
1−10−6
1−10−7
Appendix
Table 2: Fixed pipeline parameters shared by every preset unless a campaign sweeps them, and the reference estimator’s settings. The candidates’ inlier and ratio-test thresholds are the tuned parameters of Table 1 in Part I and are tuned again on the tuning cases in Part II ( Section 7.1 ); the reference estimator’s are tuned the same way at its own settings ( Section 7.2 ). The downsampling ratio and the feature budget are those of the matched operating point of Section 4.7 in every Part I ranking campaign, and points of the same budget grid in Part II’s two regimes ( Section 7.1 ). The mask margin m scales every box about its center by 1+m before it is rasterized.
Reference frame
a frame of the hover chosen by the case selection
Candidate references
every 900 frames, clear of the hover’s ends, with 3600 frames after them
Selection offsets
1 , 2 , 5 , 11 , 22 frames
Displaced
farther than 0.1 median object width
Moving together
within 0.1 median object width or 25 % of the displacement, whichever is larger
Regimes
offline: ratio 0.50 , 2000 features; real-time: ratio 0.18 , 1000 features
Inlier threshold
in pixels of the downsampled frame, both regimes
Appendix
Table 3: Part II protocol constants.
Descriptor
Candidates
Pearson r
Spearman ρ
Static foreground
831
0.972
0.955
Moving (displaced) foreground
831
0.974
0.953
Coherently moving foreground
831
0.970
0.933
Coherence
726
0.898
0.892
Stationarity
831
0.926
0.918
Cell agreement
Candidates
Share
Appendix
Table 3: The survey’s 831 candidate reference frames against the published vehicle trajectories of the same hovers, which the survey never reads. Each descriptor is recomputed from the trajectories with the survey’s own definitions (ground coordinates, thresholds in median vehicle widths, the survey’s offsets). Areas are in square meters there and frame fractions in the survey, so agreement is stated as correlation. Cell agreement is the share of candidates placed in the same moving × coherence cell by both, among those the survey placed, by their margin from the split boundaries; the selection takes the two of largest margin per cell.
We propose a new unsupervised framework for online video stabilization. Unlike methods based on deep learning that require paired stable and unstable datasets, our approach instantiates the classical stabilization pipeline with three stages and incorporates a multithreaded buffering mechanism. This design addresses three longstanding challenges in end-to-end learning: limited data, poor controllability, and inefficiency on hardware with constrained resources. Existing benchmarks focus mainly on handheld videos with a forward view in visible light, which restricts the applicability of stabilization to domains such as UAV nighttime remote sensing. To fill this gap, we introduce a new multimodal UAV aerial video dataset (UAV-Test). Experiments show that our method consistently outperforms state-of-the-art online stabilizers in both quantitative metrics and visual quality, while achieving performance comparable to offline methods.
Tao Liu, Gang Wan, Kan Ren +1
School of Electronic and Optical Engineering, Nanjing University of Science and Technology · College of Geoexploration Science and Technology, Jilin University
The rapid movements and agile maneuvers of unmanned aerial vehicles (UAVs) induce significant observational challenges for multi-object tracking (MOT). However, existing UAV-perspective MOT benchmarks often lack these complexities, featuring predominantly predictable camera dynamics and linear motion patterns. To address this gap, we introduce DynUAV, a new benchmark for dynamic UAV-perspective MOT, characterized by intense ego-motion and the resulting complex apparent trajectories. The benchmark comprises 42 video sequences with over 1.7 million bounding box annotations, covering vehicles, pedestrians, and specialized industrial categories such as excavators, bulldozers and cranes. Compared to existing benchmarks, DynUAV introduces substantial challenges arising from ego-motion, including drastic scale changes and viewpoint changes, as well as motion blur. Comprehensive evaluations of state-of-the-art trackers on DynUAV reveal their limitations, particularly in managing the intertwined challenges of detection and association under such dynamic conditions, thereby establishing DynUAV as a rigorous benchmark. We anticipate that DynUAV will serve as a demanding testbed to spur progress in real-world UAV-perspective MOT, and we will make all resources available at link.
Jingtao Ye, Kexin Zhang, Xunchi Ma +6
Xidian University, School of Computer Science and Technology, China
Continuous 6-DoF pose estimation is essential for autonomous UAV operations. Yet, existing visual odometry and SLAM methods accumulate drift and yield only relative, up-to-scale trajectories. Single-frame geo-localization, in turn, discards temporal continuity and remains too slow for real-time use. We present OrthoTrack, a training-free system that estimates continuous 6-DoF UAV trajectories using only publicly available orthophotos and surface models as a map prior. OrthoTrack matches keyframes against the orthophoto and lifts correspondences to metric 3D via the surface model. It then propagates these map-anchored correspondences to intermediate frames with optical flow, producing absolute, metrically scaled poses at every frame without GPS or post-hoc alignment. We also introduce the MovingDrone Dataset, a large-scale benchmark pairing photorealistic UAV sequences with dense 6-DoF ground truth and co-registered multi-modal geodata including multi-temporal orthophotos. On MovingDrone and real-world benchmarks, OrthoTrack runs in real time on a single GPU. It outperforms all baselines by a large margin, even those receiving oracle scale and alignment. By relying on publicly available geodata, OrthoTrack enables deployment to new regions without site-specific adaptation.
Oussema Dhaouadi, Zuria Bauer, Johannes Michael Meier +3
ETH Zurich · TU Munich · University of Cambridge +2