cs.CVFeb 23, 2026
SaveSegmentation-Guided Homography Estimation for Long-Term Planar Tracking
Organizations: Visual Recognition Group, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic
Abstract
Recent state-of-the-art visual trackers produce high quality and long-term-stable segmentation masks. We propose to leverage these strengths for planar object tracking, in which the goal is to estimate a precise 8-degrees-of-freedom homography pose, a geometric representation not estimated by segmentation trackers. We present SAM-H - a planar object tracker that estimates homographies from segmentation mask contours via a training-free pipeline. When SAM-H is applied to masks from SAM 2, it sets a new state-of-the-art performance on the challenging PlanarTrack benchmark by a large margin, +18.4pp on the p@5 metric. We further show that segmentation-based and correspondence-based homography estimation are complementary, and propose WOFTSAM, which out-performs all prior methods on both PlanarTrack and POT-210. We also provide precise re-annotations of PlanarTrack initial poses, enabling more accurate benchmarking in the high-precision p@5 metric. The code and the re-annotations are available at https://github.com/serycjon/WOFTSAM
Figures & tables
(a)
(b)
(c)
(d)
(e)
(f)
(g)
Figure 2: Overview of the SAM-H procedure for estimating homography from a segmentation tracker output. First, corners of the SAM 2 [ 21 ] mask are robustly extracted via intersection of Hough lines. Next, a symmetry disambiguation process based on a motion model and the target appearance matches the corners with the initial frame and a homography is estimated.
Figure 3: Overview of the proposed WOFTSAM planar tracker. Like in WOFT [ 22 ] , tracking ① consists of image pre-warping with the previous frame homography and homography estimation via the Weighted Flow Homography (WFH) [ 22 ] module. If the homography estimation fails (detected by correspondence support set being small), WOFTSAM does a re-detection step ②, in which the SAM-H output acts as the pre-warping homography. If this fails too, WOFTSAM falls back ③ to .
| POT-210 | PT TST | PT TST re-annot | ||||
| method | p@5 | p@15 | p@5 | p@15 | p@5 | p@15 |
| SIFT [ 17 , 14 ] | 65.8 | 69.6 | 17.6 | 26.7 | – | – |
| LISRD [ 20 , 14 ] | 68.3 | 79.2 | 22.0 | 37.2 | – | – |
| HDN [ 26 ] | 70.9 | 92.4 | 26.6 | 56.5 | – | – |
| CGN [ 13 ] | 78.5 | 93.3 | – | – | – | – |
| HVC-Net [ 27 ] | 91.4 | 95.8 | 42.0 | 63.1 | 48.8 ( +6.8 ) | 64.3 ( +1.2 ) |
Table 1: Precision on the POT-210 [ 14 ] benchmark with improved ground-truth from [ 22 ] and on PlanarTrack TST [ 15 ] (PT TST ). Higher is better. The proposed SAM-H tracker sets a new state-of-the-art on PlanarTrack because of its ability to track robustly, including unconventional targets present in parts of the benchmark. Thanks to precise correspondence-based homography estimation and the re-detection ability provided by SAM-H , the WOFTSAM sets a new state-of-the-art on POT-210 and out-performs all prior methods on PlanarTrack. The last two columns show the effect of precise re-annotation of the initial PT TST poses as described in Section 4.4 .
Figure 4: POT-210 [ 14 ] overall ( All ) and per-attribute results. Precision measured on 5 px and 15 px alignment error thresholds on the re-annotated GT [ 22 ] . Although SAM-H is the least precise of the shown methods, using it as a robust re-detection mechanism in the proposed WOFTSAM outperforms the WOFT [ 22 ] baseline, almost halving its failure rate on the 15 px threshold. The improvement is particularly high on the sequences where re-detection is needed — blur , occlusion , and unconstrained — without decreasing performance on the rest of the benchmark.
Figure 7: PlanarTrack [ 15 ] examples, where the correspondence-based WOFTSAM ( dark orange ) significantly differ from the segmentation-based SAM-H ( light orange ). SAM 2 mask shown as a green overlay, frames are cropped around the object. Top: While SAM and consequently SAM-H tracks the mirror surface as annotated in the dataset, WOFTSAM tracks a different planar surface (the doors and part of the adjacent wall) reflected in the mirror. Bottom: Two failure modes of SAM-H . (i) SAM 2 segments correctly, but the homography estimation is incorrect due to an occlusion ( middle image ). (ii) SAM 2 “spills” beyond the prompted surface ( last two images ).
| method | p@5 | p@15 | p@50 |
|---|---|---|---|
| WOFTSAM | 67.0 | 89.9 | 95.1 |
| SAM-H | 54.2 | 81.2 | 89.9 |
Table 2: MPOT-3K [ 28 ] evaluation. The p@5 and p@15 results are provided for completeness, the p@50 is more representative due to rough semi-automatic GT annotation of the dataset.
| POT-210 | PT TST | |||
|---|---|---|---|---|
| p@5 | p@15 | p@5 | p@15 | |
| 10% | 91.9 | 97.3 | 50.2 | 75.0 |
| 20% | 91.9 | 97.5 | 51.5 | 77.2 |
| 30% | 91.6 | 97.7 | 52.0 | 77.6 |
Table 5: WOFTSAM sensitivity to the failure detection inlier threshold . The results are stable over a wide range of values. In PT TST , the results slightly improve when increasing the threshold, consistently with the WOFTSAM vs SAM-H results. Even with the worst-performing 10% threshold, WOFTSAM out-performs the second-best WOFT by a large margin (+10.2pp).
Figure 10: An example where SAM-H ( light orange ) fails during an occlusion. It fits the homography to the visible part of the object (SAM 2 mask shown as a green overlay). The correspondence-based WOFTSAM ( dark orange ) continues tracking the full object. In the POT-210 [ 14 ] dataset, most occlusions are of this type.
Figure 11: Examples of SAM 2 [ 21 ] output on selected PlanarTrack [ 15 ] sequences. The images were cropped around the target from the original full-size frames. The initial frames shown on top with the ground truth pose in red . The middle row shows the selected frame with SAM 2 segmentation overlays in green . The bottom row shows the frame without overlays for better idea of the input data. The first three examples, ( Figs. 11(a) , 11(b) and 11(c) ), are difficult to track with correspondence-based methods, but are segmented well by SAM 2. In Fig. 11(d) , the segmentation “spills” outside the intended target, but gets corrected in the SAM-H Hough line detection step. Similar situation, but not recoverable by SAM-H is shown in Fig. 11(e) . Segmentation works well during partial occlusions, but recovering the full pose based only on the mask is not always possible ( Fig. 11(f) ). Distractors sometimes cause SAM 2 tracking failure ( Fig. 11(g) ).
Figure S1 : Alignment Error success plots on POT-210 [ 14 ] , with precise GT re-annotation from [ 22 ] . The results are shown on the whole dataset ( All ) and also split by the POT-210 challenge types. Thanks to the proposed SAM-H -based re-detection scheme, WOFTSAM performs better than the WOFT baseline. The p@5 and p@15 metric thresholds are highlighted with dotted lines.
Figure S2 : POT-210 [ 14 ] examples, where outputs of WOFTSAM ( dark orange ) significantly differ from the outputs of SAM-H ( light orange ) or WOFT ( blue ). SAM 2 mask shown as a green overlay, frames are cropped around the object. Top: SAM-H homography is incorrect due to occlusion with a linear boundary. WOFT and WOFTSAM outputs overlap. Frames of V30_5 . Bottom: WOFT switches to tracking an incorrect target during occlusion ( middle image ). SAM-H stays on the correct object, although with an incorrect homography during the occlusion. WOFTSAM stays on the correct target thanks to the robust re-detection ( last image ). WOFT and WOFTSAM outputs overlap on the first two images. Frames of V10_5 .
Figure S3 : Examples of our PlanarTrack TST [ 15 ] re-annotation on the initial frames. The original GT ( red ) does not copy the object boundaries precisely, while the new improved GT ( cyan ) aligns with the target.
Figure S4 : Error histograms of the original PlanarTrack TST [ 15 ] initial frame annotations compared to our re-annotation. Top: alignment error , capturing the annotation difference over all the four control points (RMSE). Bottom: euclidean distance of individual control point positions, with y-axis scaled logarithmically.
| voting window | smoothing | line-point px | line-point ∘ | PT TST p@5 | PT TST p@15 |
|---|---|---|---|---|---|
| 10 | 4 | 15 | 15 | 62.4 | 80.5 |
| 5 | 62.4 ( ) | 80.5 ( ) | |||
| 20 | 61.9 ( ) | 79.8 ( ) | |||
| 2 | 62.2 ( ) | 80.1 ( ) | |||
| 8 | 60.8 ( ) | 78.6 ( ) | |||
| 10 | 62.5 ( ) | 80.5 ( ) |
Table S1 : Hough hyperparameters evaluated on the PlanarTrack TST dataset with re-annotated init frames – SAM-H is stable in a wide range. First row is the default SAM-H , other rows alter one hyperparameter. Computed on newer stack than the main paper results (NVIDIA A100 GPU, AMD EPYC 7543 CPU, CUDA 12.6, CuDNN 9.5.0, PyTorch 2.7.1), leading to a small difference in the results (+0.4pp on both PT thresholds for SAM-H ). The method itself did not change.
| method | p@5 | p@15 |
|---|---|---|
| SAM-H | 62.4 ( ) | 80.5 ( ) |
| SAM-H no DINO | 53.7 | 68.0 |
| min-area rectangle | 13.6 ( ) | 39.2 ( ) |
| RDP approximation | 39.1 ( ) | 70.1 ( ) |
Table S2 : Symmetry disambiguation and mask-to-quad baselines evaluated on the PlanarTrack TST dataset with re-annotated init frames. The proposed SAM-H achieves significantly better performance than the baselines. Computed on newer stack than the main paper results (NVIDIA A100 GPU, AMD EPYC 7543 CPU, CUDA 12.6, CuDNN 9.5.0, PyTorch 2.7.1), leading to a small difference in the results (+0.4pp on both PT thresholds for SAM-H ). The method itself did not change.
More in this topic
Visual Object Tracking
Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.
See all 175 papers