TAPNext++: What's Next for Tracking Any Point (TAP)?

Sebastian Jung; Artem Zholus; Martin Sundermeyer; Carl Doersch; Ross Goroshin; David Joseph Tan; Sarath Chandar; Rudolph Triebel

cs.CVApr 12, 2026

TAPNext++: What's Next for Tracking Any Point (TAP)?

Authors: Sebastian Jung, Artem Zholus, Martin Sundermeyer, Carl Doersch, Ross Goroshin, David Joseph Tan, Sarath Chandar, Rudolph Triebel,

Organizations: German Aerospace Center (DLR) · Karlsruhe Institute of Technology (KIT) · Mila - Quebec AI Institute · Chandar Research Lab ·

Abstract

Tracking-Any-Point (TAP) models aim to track any point through a video which is a crucial task in AR/XR and robotics applications. The recently introduced TAPNext approach proposes an end-to-end, recurrent transformer architecture to track points frame-by-frame in a purely online fashion -- demonstrating competitive performance at minimal latency. However, we show that TAPNext struggles with longer video sequences and also frequently fails to re-detect query points that reappear after being occluded or leaving the frame. In this work, we present TAPNext++, a model that tracks points in sequences that are orders of magnitude longer while preserving the low memory and compute footprint of the architecture. We train the recurrent video transformer using several data-driven solutions, including training on long 1024-frame sequences enabled by sequence parallelism techniques. We highlight that re-detection performance is a blind spot in the current literature and introduce a new metric, Re-Detection Average Jaccard ( $AJ_{RD}$ ), to explicitly evaluate tracking on re-appearing points. To improve re-detection of points, we introduce tailored geometric augmentations, such as periodic roll that simulates point re-entries, and supervising occluded points. We demonstrate that recurrent transformers can be substantially improved for point tracking and set a new state-of-the-art on multiple benchmarks. Model and code can be found at https://tap-next-plus-plus.github.io.

Figures & tables

Figure 2: Qualitative Comparison on Challenging Re-entry Scenarios. We compare TAPNext++ with prior methods on two sequences featuring point re-entries at varying locations. Left: An object leaves the frame on the right and re-enters from the left. BootsTAPNext and Track-On fail to re-detect the object upon re-entry. In contrast, CoTracker3 and TAPNext++ successfully relocate the points. However, CoTracker3 fails to track static points on the texture-less table, which TAPNext++ tracks correctly. The variant TAPNext++ without Roll also struggles with immediate re-detection, highlighting the importance of roll augmentation for re-entry robustness. Right: Three objects are juggled, repeatedly leaving and re-entering the frame. Point trajectories for the last 8 frames are visualized as traces for clarity. TAPNext++ with roll is the only method that robustly tracks all objects through these challenging dynamics, all competing methods fail.

Figure 3: Distributed Parallel Scan during Training. On the left, the TAPNext architecture with its SSM and ViT Blocks is shown. Right, example of information flow during multi-gpu training using a distributed parallel scan. Note that information is only passed between the SSM Blocks of the GPUs and not the ViT Blocks. Only inter-GPU communication is shown, intra-GPU parallel scans are not shown for simplicity. For a detailed figure of the TAPNext architecture we refer to [ 30 ] .

Figure 4: Roll Augmentation. Videos are rotated and translated smoothly over time, wrapping around the image with a gap. Ground-truth points are visualized.

Figure 5: Long-Sequence Stability. Mean Pixel Error of 16x16 points after repeatedly inferring a static image through TAPNext variants. Finetuning on 1024-frame videos from PointOdyssey most effectively extends the memory longevity.

Figure 6:

\text{AJ}_{\mathrm{RD}}

Comparison. Re-Detection Average Jaccard (

\text{AJ}_{\mathrm{RD}}

) for different

d_{\text{min}}

values on RoboTAP. TAPNext++ with roll augmentations preserves AJ after long periods of undetectability. The window-based CoTracker3 exhibits a drop in

\text{AJ}_{\mathrm{RD}}

following a sequence of 256 undetectable frames.

Figure 7: Long-Term Dynamic Tracking Example. We compare state-of-the-art online trackers on tracking clock hands, visualizing a trace of the last 6 frames. While the original BootsTAPNext, Track-On and CoTracker3 fail at tracking both clock hands for the full duration of the video, TAPNext++ tracks both until the end. CoTracker3 and Track-On are only able to track the hour hand due to its fine structure, additionally CoTracker3 loses the hour hand at the end of the video.

#	Method	Res.	PointOdyssey				DAVIS			RGB-Stacking			RoboTAP				Kinetics			Mean
			$\delta^{avg}$ $\uparrow$	Survival $\uparrow$	MTE $\downarrow$	$\text{AJ}_{\mathrm{RD}}$ $\uparrow$	AJ $\uparrow$	$\delta^{avg}$ $\uparrow$	OA $\uparrow$	AJ $\uparrow$	$\delta^{avg}$ $\uparrow$	OA $\uparrow$	AJ $\uparrow$	$\delta^{avg}$ $\uparrow$	OA $\uparrow$	$\text{AJ}_{\mathrm{RD}}$ $\uparrow$	AJ $\uparrow$	$\delta^{avg}$ $\uparrow$	OA $\uparrow$	$\delta^{avg}$ $\uparrow$
1	PIPs++	$256\times 256$	21.5	38.1	46.1	–	–	63.4	–	–	58.5	–	–	63.0	–	–	–	–	–	51.6
2	CoTracker2	$384\times 512$	30.2	55.2	215.0	3.9	62.2	75.7	89.3	67.4	78.9	85.2	58.6	70.6	87.0	33.3	48.8	64.5	85.8	64.0
3	CoTracker3 (Online)	$384\times 512$	44.5	56.3	20.7	21.8	63.8	76.3	90.2	71.7	83.6	90.2	66.4	78.8	90.8	51.8	55.8	68.5	88.3	70.2
4	Track-On	$384\times 512$	35.4	47.5	33.5	9.9	65.0	78.0	90.8	71.4	85.2	91.7	63.5	76.4	89.4	44.4	53.9	67.3	87.8	68.5
5	BootsTAPNext-B	$256\times 256$	9.9	13.0	92.1	0.6	65.2	78.5	91.2	66.2	78.3	86.8	62.6	74.3	88.4	47.1	57.3	70.6	87.4	62.3

Table 1: Online Tracking Comparison. We report results on multiple point tracking benchmarks and compare against current state-of-the-art online trackers.

Query Points	Model	Average FPS $\uparrow$	Latency (ms) $\downarrow$
	LocoTrack-B (window)	452	$2210$
	CoTracker3 (online)	102	$80$
	Track-On (frame)	28	$36$
256	TAPNext (frame)	$189$	$5.29$
	TAPNext++ (frame)	$193$	$\mathbf{5.18}$
	TAPNext (window)	$480$	$66$

Table 2: Inference Speed. Speed comparison of TAPNext++ to TAPNext, LocoTrack-B (

256\times 256

), online CoTracker3 and Track-On running on Nvidia H100 GPUs. The latency metric is defined as the maximum (worst case) time between passing a frame to the model and receiving predicted points and it includes the time it takes to fill and process the initial frame buffer. (frame) indicates per frame inference, (window) is when we track with non-overlapping chunks of 32 frames. All models are implemented in PyTorch with TAPNext++ relying on FlashAttention3 [ 25 ] .

#	Aug.		PointOdyssey				DAVIS			RoboTAP				Kinetics
	Roll	Aspect	$\delta^{avg}$ $\uparrow$	Survival $\uparrow$	MTE $\downarrow$	$\text{AJ}_{\mathrm{RD}}$ $\uparrow$	AJ $\uparrow$	$\delta^{avg}$ $\uparrow$	OA $\uparrow$	AJ $\uparrow$	$\delta^{avg}$ $\uparrow$	OA $\uparrow$	$\text{AJ}_{\mathrm{RD}}$ $\uparrow$	AJ $\uparrow$	$\delta^{avg}$ $\uparrow$	OA $\uparrow$
1	$\checkmark$	$\checkmark$	49.3	63.5	17.4	20.4	66.0	79.5	91.9	59.8	74.2	88.4	51.1	53.6	68.3	88.5
2			51.4	68.3	15.6	22.8	64.7	78.4	91.6	59.3	73.7	88.0	50.7	53.0	68.0	88.2
3		$\checkmark$	52.0	68.6	16.4	22.3	65.3	79.1	91.4	59.2	73.9	88.0	50.0	52.6	67.8	87.8
4	$\checkmark$		49.7	65.0	15.8	20.7	64.7	78.8	91.4	60.4	74.7	88.7	52.0	53.7	68.4	88.6

Table 3: Impact of Augmentations. Data augmentation while finetuning TAPNext on PointOdyssey (PO) impacts generalizability. While less augmentations lead to higher results on PO, using roll and aspect ratio augmentations lead to overall better results across other datasets.

#	Dataset			PointOdyssey				DAVIS			RoboTAP				Kinetics
	PO	Kub1024	DynHumans	$\delta^{avg}$ $\uparrow$	Survival $\uparrow$	MTE $\downarrow$	$\text{AJ}_{\mathrm{RD}}$ $\uparrow$	AJ $\uparrow$	$\delta^{avg}$ $\uparrow$	OA $\uparrow$	AJ $\uparrow$	$\delta^{avg}$ $\uparrow$	OA $\uparrow$	$\text{AJ}_{\mathrm{RD}}$ $\uparrow$	AJ $\uparrow$	$\delta^{avg}$ $\uparrow$	OA $\uparrow$
1	$\checkmark$	$\checkmark$	$\checkmark$	52.6	67.9	13.4	23.3	66.6	79.9	92.1	61.1	75.2	89.6	52.4	54.4	68.7	89.0
2	$\checkmark$	$\checkmark$		51.8	71.0	14.4	23.3	65.6	79.0	92.0	61.1	75.4	89.0	52.6	53.9	68.4	88.7
3	$\checkmark$			49.3	63.5	17.4	20.4	66.0	79.5	91.9	59.8	74.2	88.4	51.1	53.6	68.3	88.5

Table 4: Multi-Dataset Finetuning. Incorporating multiple datasets in the finetuning helps generalizability while also improving the results on PointOdyssey (PO) compared to training on PO alone.

Method	$\delta^{avg}$ $\uparrow$	Survival $\uparrow$	MTE $\downarrow$
CoTracker3	44.5	56.3	20.7
Track-On	35.4	47.5	33.5
BootsTAPNext-B (Baseline)	9.9	13.0	92.1
Ours (w/ PO)	52.6	67.9	13.4
Ours (w/o PO)	37.3	51.3	26.4

Table 5: Training without PointOdyssey. Evaluation results on PointOdyssey (PO) with and without including PO in the training mix.

Figure 8: LRU State Norm Growth on re-feeding a Single Image. BootsTAPNext shows unstable LRU state after 100 frames. TAPNext++ has a monotonically growing state.

Figure 9: Eigenvalue Distribution. Distribution of eigenvalues $a$ . Almost no change can be observed between BootsTAPNext and TAPNext++.

Point Tracking

Point tracking follows selected or dense image points through video, enabling motion analysis, 3D reconstruction, and precise spatiotemporal understanding.

See all 35 papers

TAPFormer: Robust Arbitrary Point Tracking via Transient Asynchronous Fusion of Frames and Events
Mar 5, 2026Jiaxiong Liu, Zhen Tan, Jinpu Zhang +4Point Tracking Multimodal Sensor Fusion
National University of Defense Technology · Hunan University
TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views
Sep 1, 2026Skanda Koppula, Frano Rajic, Abdullah Faiz Ur Rahman +9Point Tracking 3D Point Tracking
Google DeepMind · University College London · ETH Zürich +1
TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations
Sep 24, 2026Ayush Jain, Sreeharsha Paruchuri, Ishita Gupta +5Point Tracking 3D Point Tracking
Carnegie Mellon University · Meta

Mar 5, 2026J/K move · Enter open · S save

TAPFormer: Robust Arbitrary Point Tracking via Transient Asynchronous Fusion of Frames and Events

Jiaxiong Liu, Zhen Tan, Jinpu Zhang, Yi Zhou+3

National University of Defense Technology · Hunan University

Point Tracking Multimodal Sensor Fusion Event-Based Vision

Open paper AskPDF arXiv

Tracking any point (TAP) is a fundamental yet challenging task in computer vision, requiring high precision and long-term motion reasoning. Recent attempts to combine RGB frames and event streams have shown promise, yet they typically rely on synchronous or non-adaptive fusion, leading to temporal misalignment and severe degradation when one modality fails. We introduce TAPFormer, a transformer-based framework that performs asynchronous temporal-consistent fusion of frames and events for robust and high-frequency arbitrary point tracking. Our key innovation is a Transient Asynchronous Fusion (TAF) mechanism, which explicitly models the temporal evolution between discrete frames through continuous event updates, bridging the gap between low-rate frames and high-rate events. In addition, a Cross-modal Locally Weighted Fusion (CLWF) module adaptively adjusts spatial attention according to modality reliability, yielding stable and discriminative features even under blur or low light. To evaluate our approach under realistic conditions, we construct a novel real-world frame-event TAP dataset under diverse illumination and motion conditions. Our method outperforms existing point trackers, achieving a 28.2% improvement in average pixel error within threshold. Moreover, on standard point tracking benchmarks, our tracker consistently achieves the best performance. Project website: tapformer.github.io

Abstract

Point Tracking

Explore similar work

TAPFormer: Robust Arbitrary Point Tracking via Transient Asynchronous Fusion of Frames and Events

TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views

TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations