TAP-Out: Tracking Any Point in 360° via Outpainting
Organizations: Department of Computer Science, University of York, York, United Kingdom
Abstract
Tracking points that leave the camera's field of view (FoV) requires reasoning about unobserved scene dynamics, a fundamentally different problem from tracking visible correspondences. We present TAP-Out, an offline method that makes this out-of-view (OOV) reasoning explicit by leveraging generative video outpainting as a world-model prior. Given a narrow-FoV input video, TAP-Out outpaints a complete 360° equirectangular video, locates target objects within the panoramic scene via video object segmentation, and synthesises object-centred perspective crops on which standard in-frame trackers operate. This decoupled design composes off-the-shelf foundation models without requiring task-specific architectural modifications. On the TAPVid360 benchmark, TAP-Out achieves state-of-the-art performance, improving OOV tracking accuracy by approximately 3x over the best prior method. We demonstrate its flexibility on out-of-frame object segmentation and detection and we provide detailed diagnostic analyses across motion regimes, static versus dynamic content, and hallucination quality, characterising when and why the approach succeeds or fails. Code is publicly available at https://github.com/finlay-hudson/TAP-Out
Figures & tables
| Method | all | if | oof | all | if | oof |
|---|---|---|---|---|---|---|
| TAPNext [ Zholus et al.(2025)Zholus, Doersch, Yang, Koppula, Patraucean, He, Rocco, Sajjadi, Chandar, and Goroshin ] | 0.0075 0.0077 | 0.0178 0.0180 | 0.0004 0.0010 | 52.6330 18.4256 | 37.2676 19.6733 | 62.6947 19.8463 |
| TAPIR [ Doersch et al.(2023)Doersch, Yang, Vecerik, and Gokay ] | 0.0096 0.0096 | 0.0229 0.0225 | 0.0003 0.0006 | 50.7070 18.4413 | 35.0867 21.8100 | 60.9705 17.8270 |
| BootsTAPIR [ Doersch et al.(2024)Doersch, Luc, Yang, Gokay, Koppula, Gupta, Heyward, Rocco, Goroshin, Carreira, and Zisserman ] | 0.0106 0.0106 | 0.0254 0.251 | 0.0005 0.0011 | 49.1382 18.5384 | 34.4510 21.4859 | 58.6180 18.2366 |
| TAPIP3D [ Zhang et al.(2025)Zhang, Ke, Harley, and Fragkiadaki ] | 0.2514 0.1606 | 0.4773 0.2429 | 0.0897 0.1301 | 35.9088 26.7414 | 22.9776 23.0838 | 44.8629 32.1248 |
| CoTracker3 (offline) [ Karaev et al.(2024b)Karaev, Rocco, Graham, Neverova, Vedaldi, and Rupprecht ] | 0.2375 0.0899 | 0.5516 0.1588 | 0.0146 0.0208 | 38.2831 21.9041 | 18.4400 22.2117 | 51.6813 24.4290 |
| CoTracker360 [ Hudson et al.(2025)Hudson, Gardner, and Smith ] | 0.2259 0.1059 | 0.3925 0.1326 | 0.1062 0.0984 | 8.6054 7.6706 | 4.1213 4.8057 | 11.3619 9.8862 |
| Regime | Low ( \dg – \dg ) | Medium ( \dg – \dg ) | High ( \dg +) |
|---|---|---|---|
| % Points in Dataset | 0.8% | 59.6% | 39.6% |
| TAP-Out oof | 0.90 | 0.45 | 0.21 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | all | if | oof | all | if | oof |
|---|---|---|---|---|---|---|
| TAPNext [ Zholus et al.(2025)Zholus, Doersch, Yang, Koppula, Patraucean, He, Rocco, Sajjadi, Chandar, and Goroshin ] | 0.0082 (0.0082) | 0.0190 (0.0191) | 0.0004 (0.0004) | 51.98 (51.98) | 36.58 (36.59) | 62.45 (62.46) |
| TAPIR [ Doersch et al.(2023)Doersch, Yang, Vecerik, and Gokay ] | 0.0106 (0.0106) | 0.0250 (0.0251) | 0.0003 (0.0003) | 49.83 (49.81) | 33.87 (33.88) | 60.53 (60.52) |
| BootsTAPIR [ Doersch et al.(2024)Doersch, Luc, Yang, Gokay, Koppula, Gupta, Heyward, Rocco, Goroshin, Carreira, and Zisserman ] | 0.0125 (0.0126) | 0.0291 (0.0293) | 0.0005 (0.0005) | 48.37 (48.36) | 33.30 (33.32) | 58.38 (58.38) |
| TAPIP3D [ Zhang et al.(2025)Zhang, Ke, Harley, and Fragkiadaki ] | 0.2449 (0.2476) | 0.4685 (0.4698) | 0.0817 (0.0850) | 36.44 (36.44) | 23.08 (23.32) | 45.90 (45.80) |
| SpatialTracker [ Xiao et al.(2024)Xiao, Wang, Zhang, Xue, Peng, Shen, and Zhou ] | 0.2235 (0.2239) | 0.4889 (0.4893) | 0.0300 (0.0303) | 38.90 (38.88) | 22.16 (22.16) | 50.45 (50.44) |
| CoTracker3 (offline) [ Karaev et al.(2024b)Karaev, Rocco, Graham, Neverova, Vedaldi, and Rupprecht ] | 0.2444 (0.2435) | 0.5601 (0.5588) | 0.0158 (0.0158) | 37.40 (37.43) | 17.60 (17.64) | 50.94 (50.98) |
| Generative Prior | 2D Tracker Backbone | (OOV Acc.) | (Mean Error) |
|---|---|---|---|
| Argus | CoTracker3 (Ours, Default) | 0.339 | 5.08 ∘ |
| Argus | BootsTAPIR | 0.298 | 5.85 ∘ |
Point Tracking
Point tracking follows selected or dense image points through video, enabling motion analysis, 3D reconstruction, and precise spatiotemporal understanding.
See all 39 papers