TAPFormer: Robust Arbitrary Point Tracking via Transient Asynchronous Fusion of Frames and Events

Jiaxiong Liu; Zhen Tan; Jinpu Zhang; Yi Zhou; Hui Shen; Xieyuanli Chen; Dewen Hu

cs.CVMar 5, 2026

TAPFormer: Robust Arbitrary Point Tracking via Transient Asynchronous Fusion of Frames and Events

Authors: Jiaxiong Liu, Zhen Tan, Jinpu Zhang, Yi Zhou, Hui Shen, Xieyuanli Chen, Dewen Hu

Organizations: National University of Defense Technology · Hunan University

Abstract

Tracking any point (TAP) is a fundamental yet challenging task in computer vision, requiring high precision and long-term motion reasoning. Recent attempts to combine RGB frames and event streams have shown promise, yet they typically rely on synchronous or non-adaptive fusion, leading to temporal misalignment and severe degradation when one modality fails. We introduce TAPFormer, a transformer-based framework that performs asynchronous temporal-consistent fusion of frames and events for robust and high-frequency arbitrary point tracking. Our key innovation is a Transient Asynchronous Fusion (TAF) mechanism, which explicitly models the temporal evolution between discrete frames through continuous event updates, bridging the gap between low-rate frames and high-rate events. In addition, a Cross-modal Locally Weighted Fusion (CLWF) module adaptively adjusts spatial attention according to modality reliability, yielding stable and discriminative features even under blur or low light. To evaluate our approach under realistic conditions, we construct a novel real-world frame-event TAP dataset under diverse illumination and motion conditions. Our method outperforms existing point trackers, achieving a 28.2% improvement in average pixel error within threshold. Moreover, on standard point tracking benchmarks, our tracker consistently achieves the best performance. Project website: tapformer.github.io

Figures & tables

Figure 1: A qualitative comparison of tracking performance in our dataset shows that (a) frame-based tracking suffers from insufficient temporal information, and (b) event-based tracking fails to capture fine spatial details. In contrast, our (c) fusion approach can recover long-term, high-accuracy point trajectories. The rightmost color bar visualizes the per-point tracking error (in pixels).

Figure 2: TAPFormer overview. (a) The overall framework: frames and events are fused by the transient asynchronous fusion mechanism and Cross-Modal Local Weighted Fusion modules to produce high-frequency transient features, refined by temporal attention and decoded into multi-scale fusion features. The resulting features, together with the initial query points position q , are fed into a transformer-based optimization module to iteratively predict tracking trajectories x and occlusion states $v$ . $M$ denotes the number of iterations. (b) The fusion network: image and event tokens are integrated by local weighted cross-attention to construct and update transient representations.

Method	Train Data	Input	Time $\downarrow$	InivTAP			DrivTAP			Mean $\delta_{\text{avg}}^{\text{vis}}$ $\uparrow$
Method	Train Data	Input	Time $\downarrow$	AJ $\uparrow$	$\delta_{\text{avg}}^{\text{vis}}\uparrow$	OA $\uparrow$	AJ $\uparrow$	$\delta_{\text{avg}}^{\text{vis}}\uparrow$	OA $\uparrow$	Mean $\delta_{\text{avg}}^{\text{vis}}$ $\uparrow$
PIPs++ [ 57 ]	PO	Frame	21.1	–	31.2	–	–	37.8	–	34.5
Chrono [ 26 ]	Kub	Frame	-	31.2	44.7	79.8	15.8	30.8	68.1	37.8
CoTracker3 [ 24 ]	Kub	Frame	19.0	41.8	53.2	72.8	37.1	46.5	95.4	49.9
TAPFormer-F (Ours)	FastKub	Frame	16.5	44.6	54.5	74.7	36.6	46.7	93.8	50.6
E2Vid [ 43 ] -PIPs++ [ 57 ]	PO	Event	21.4	–	14.3	–	–	30.4	–	22.4

Table 1: Comparison with state-of-the-art arbitrary point tracking methods. Training data: PO (Point Odyssey), Kub (Kubric), MF (MultiFlow), FE-FastKub (Our training dataset). speed expressed as ms per time step. Gray rows indicate our methods.

Figure 3: Task 1: TAP on InivTAP . Rows show different sequences, and columns correspond to frame-based, event-based, fusion-based (ours), and ground-truth results.

Figure 4: Red boxes show fast-moving vehicles where frame-based tracking drifts; yellow boxes highlight texture-similar regions where event-based tracking fails. Our fusion-based method achieves stable and accurate tracking in both cases.

Method	Input	EDS		EC
Method	Input	FA $\uparrow$	EFA $\uparrow$	FA $\uparrow$	EFA $\uparrow$
PIPs++ [ 57 ]	Frame	75.1	63.0	82.6	82.3
CoTracker3 [ 24 ]	Frame	80.2	68.8	92.5	91.9
Chrono [ 26 ]	Frame	–	–	82.9	82.3
TAPFormer-F (Ours)	Frame	80.9	69.1	92.7	92.1
EM-ICP [ 59 ]	Event	16.1	12.0	33.7	33.4

Table 2: Comparison of tracking accuracy on EDS and EC datasets. Values are in percentages (%). The best results are bold.

Figure 5: Task 2: Feature tracking on EDS.

FE-FastKub	CLWF	TAF	MSSF	TAM	FA $\uparrow$	EFA $\uparrow$
					0.646	0.535
✓					0.701	0.585
✓	✓				0.763	0.647
✓	✓	✓			0.803	0.685
✓	✓	✓	✓		0.814	0.698
✓	✓	✓		✓	0.807	0.691

Table 3: Ablation study on the EDS dataset. FA and EFA denote feature age and expected feature age, respectively. MSSF and TAM denote multi-scale semantic features and temporal attention module, respectively.

Figure 6: PCA visualization of feature embeddings from frame-, event-, and fusion-based representations. Each color denotes a tracked point; Ellipses indicate the covariance-based dispersion of feature embeddings in the PCA space, reflecting the intra-point stability and inter-point separability of learned representations.

Method	Frame rate (FPS)
Method	75	37.5	25	18.75	12.5	9.375
CoTracker3	83.9	74.9	51.2	32.4	20.7	13.2
Ours	85.1	83.2	82.7	80.8	79.6	75.8

Table 4: Effect of frame rate on tracking performance. Results are reported on the EDS dataset using different input frame rates.

Figure 7: Frame-rate sensitivity comparison between CoTracker3 and our method. (a) shows overall performance trends with variance shading; (b–c) visualize relative performance in slow and fast motion scenarios, where color bands denote absolute performance differences (see color bar on the right).

Representation	EDS		EC
Representation	FA $\uparrow$	EFA $\uparrow$	FA $\uparrow$	EFA $\uparrow$
Event image [ 36 ]	81.4	69.5	90.8	90.2
Voxel grid [ 60 ]	82.5	69.8	92.1	91.5
Time surface [ 27 ]	82.3	70.4	93.3	92.6

Table 5: Effect of event representation on tracking performance.

Figure 8: Performance and efficiency across tracking iterations. The left y-axis reports the feature age (higher is better), while the right y-axis shows the inference throughput measured in frames per millisecond (FPS/ms). The x-axis denotes the number of trajectory refinement iterations. This dual-axis plot illustrates how tracking accuracy and computational efficiency evolve as the number of iterations increases.

Figure 9: Ground-truth trajectory correction for the peanuts light sequence. (a) Pre-correction trajectory with two displacement outliers. (b) Post-correction trajectory after smoothing, producing a stable and reliable ground truth.

Figure 10: Additional visualizations on EC and EDS dataset.

Scenario	Method	Modality	AJ $\uparrow$	$\delta_{\text{avg}}^{\text{vis}}$ $\uparrow$	OA $\uparrow$
Normal	ETAP	Event	19.7	33.6	88.8
	CoTracker3	Frame	61.3	77.1	92.1
	Ours	F + E	71.1	84.8	92.7
Fast	ETAP	Event	27.6	40.6	89.2
	CoTracker3	Frame	28.0	36.7	73.5
	Ours	F + E	39.5	52.3	95.4

Table 6: Performance comparison across different scenarios and modalities.

Method	Modalities			AUC v $\uparrow$
Method	Imu Used	Events Used	Images Used	AUC v $\uparrow$
SDEVO [ 58 ]	none	stereo	none	6.13
SuperEvent [ 3 ]	yes	stereo	stereo	6.27
Ours	yes	monocular	monocular	6.29

Table 7: Comparison with event-based SLAM methods. We compare our approach with two event-based SLAM baselines. After directly replacing the VINS-Mono frontend with ours, our method achieves the best performance.

Figure 11: Visualization of our tracking results on SLAM sequences. (a–c) show results on high-speed UAV scenarios, while (d–e) present tracking results on ground-based high-speed sequences.

Figure 12: Data collection setup. (b) Our custom synchronized capture device is mounted on the (a) vehicle platform to record DrivTAP, and (c) a DAVIS346 camera is used to collect the InivTAP dataset.

Method	Average		shapes_trans		shapes_rot		shapes_6dof		boxes_trans		boxes_rot
Method	FA $\uparrow$	EFA $\uparrow$	FA $\uparrow$	EFA $\uparrow$	FA $\uparrow$	EFA $\uparrow$	FA $\uparrow$	EAF $\uparrow$	FA $\uparrow$	EFA $\uparrow$	FA $\uparrow$	EFA $\uparrow$
PIPs++ [ 57 ]	82.6	82.3	87.1	87.1	79.0	78.8	83.7	82.7	86.3	86.1	77.0	76.9
Cotracker3 [ 24 ]	92.5	91.9	94.4	94.3	87.3	86.8	93.3	91.6	93.6	93.2	94.3	93.9
Chrono [ 26 ]	82.9	82.3	85.1	84.5	81.5	81.1	75.6	74.1	83.3	83.0	88.8	88.6
TAPFormer-F	92.7	92.1	95.2	95.1	87.8	87.4	92.8	91.1	92.9	92.4	94.7	94.4
EM-ICP [ 59 ]	33.7	33.4	40.3	40.2	32.0	32.0	24.8	24.2	35.5	35.4	35.6	34.9

Table 8: Tracking performance comparison on the EC dataset. The first four rows list frame-only methods, followed by five event-only baselines, and the last four rows present approaches fusing frames and events. Methods with a gray background denote our models. All results are reported in percentages, and the best performance in each column is highlighted in bold .

Method	Average		Peanuts Light		Rocket Earth*		Ziggy Arena		Peanuts Running
Method	FA $\uparrow$	EFA $\uparrow$	FA $\uparrow$	EFA $\uparrow$	FA $\uparrow$	EFA $\uparrow$	FA $\uparrow$	EFA $\uparrow$	FA $\uparrow$	EFA $\uparrow$
PIPs++ [ 57 ]	75.1(72.1)	63.0(60.3)	62.6(50.9)	58.8(47.9)	76.4	34.2	85.1	85.0	76.1	74.1
cotracker3 [ 24 ]	80.2(75.8)	68.8(64.7)	72.7(54.9)	68.4(51.9)	69.2	30.7	92.6	92.5	86.4	83.7
TAPFormer-F	80.9(76.4)	69.1(64.9)	74.7(56.7)	70.1(53.3)	71.5	31.5	91.8	91.8	85.6	82.8
EM-ICP [ 59 ]	16.1	12.0	8.4	7.7	29.8	15.8	15.3	14.9	10.8	9.5
HASTE [ 1 ]	9.6	6.3	8.6	7.6	16.2	8.5	8.2	5.7	5.4	3.3

Table 9: Tracking performance comparison in EDS dataset. The values in parentheses indicate the performance of the Peanuts Light sequence under ground-truth evaluation before trajectory correction. For the Average column, the value in parentheses indicates the mean score after replacing only the Peanuts Light result. Rocket Earth ∗ indicating that the ground truth is not fully accurate. All results are reported in percentages, and the best performance in each column is highlighted in bold .

Dataset	Source	Events	Final image	Fast scene	#Samples	Resolution	fps[Hz]	sample duration[s]	Annotations
Dataset	Source	Events	Final image	Fast scene	#Samples	Resolution	fps[Hz]	sample duration[s]	optical flow	TAP	depth	segmentations
TAP-Vid Kubric [ 8 ]	3D PBR	none	$\checkmark$	$\times$	$\approx 10000$	$512\times 512$	12	2	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
BlinkFlow [ 31 ]	3D PBR	synthetic	$\checkmark$	$\times$	3587	$640\times 480$	10	1	$\checkmark$	$\times$	$\checkmark$	$\checkmark$
MultiFlow [ 14 ]	2D warp	synthetic	$\times$	$\times$	12100	$512\times 384$	100	0.5	$\checkmark$	$\times$	$\times$	$\times$
EventKubric [ 16 ]	3D PBR	synthetic	$\times$	$\times$	10173	$512\times 512$	48	2	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
FE-FastKub(Ours)	3D PBR	synthetic	$\checkmark$	$\checkmark$	10953	$512\times 512$	48	2	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 10: Dataset comparison. Overview of publicly available synthetic event-based motion estimation datasets.

Figure 14: Statistics of ground-truth trajectories in InivTAP and DrivTAP. Valid coordinate ratio measures the percentage of time a queried point remains within the image boundaries. Diameter denotes the trajectory’s motion range normalized by image height. Number of segments indicates how many continuous track fragments are formed due to occlusions. Displacement represents the per-frame motion magnitude normalized by image height.

Figure 15: Annotation interface for the TAP task. We select either RGB frames or event-reconstructed frames for manual ground-truth labeling, depending on which modality provides clearer visual cues. The tool consists of three main components: a visualization panel for displaying the sequence, an annotation information panel for managing point information, and a button panel providing operation interactive controls.

Point Tracking

Point tracking follows selected or dense image points through video, enabling motion analysis, 3D reconstruction, and precise spatiotemporal understanding.

See all 35 papers

TAPNext++: What's Next for Tracking Any Point (TAP)?
Apr 12, 2026Sebastian Jung, Artem Zholus, Martin Sundermeyer +6Point Tracking Recurrent Transformers
German Aerospace Center (DLR) · Karlsruhe Institute of Technology (KIT) · Mila - Quebec AI Institute +7
Spatial Orthogonal Refinement for Robust RGB-Event Visual Object Tracking
Mar 29, 2026Dexing Huang, Shiao Wang, Fan Zhang +1RGB-Event Sensor Fusion Visual Object Tracking
School of Chemistry & Chemical Engineering, Anhui University, Hefei 230601, China · School of Computer Science and Technology, Anhui University, Hefei 230601, China
Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking
May 7, 2026Shiao Wang, Xiao Wang, Duoqing Yang +5Event-Based Vision Mixture of Experts
School of Computer Science and Technology, Anhui University, Hefei 230601, China · Beijing Institute of Technology, Beijing, China · Peng Cheng Laboratory, Shenzhen, China +2

Apr 12, 2026J/K move · Enter open · S save

TAPNext++: What's Next for Tracking Any Point (TAP)?

Sebastian Jung, Artem Zholus, Martin Sundermeyer, Carl Doersch+5

German Aerospace Center (DLR) · Karlsruhe Institute of Technology (KIT) · Mila - Quebec AI Institute+7

Point Tracking Recurrent Transformers

Open paper AskPDF arXiv

Tracking-Any-Point (TAP) models aim to track any point through a video which is a crucial task in AR/XR and robotics applications. The recently introduced TAPNext approach proposes an end-to-end, recurrent transformer architecture to track points frame-by-frame in a purely online fashion -- demonstrating competitive performance at minimal latency. However, we show that TAPNext struggles with longer video sequences and also frequently fails to re-detect query points that reappear after being occluded or leaving the frame. In this work, we present TAPNext++, a model that tracks points in sequences that are orders of magnitude longer while preserving the low memory and compute footprint of the architecture. We train the recurrent video transformer using several data-driven solutions, including training on long 1024-frame sequences enabled by sequence parallelism techniques. We highlight that re-detection performance is a blind spot in the current literature and introduce a new metric, Re-Detection Average Jaccard (

AJ_{RD}

), to explicitly evaluate tracking on re-appearing points. To improve re-detection of points, we introduce tailored geometric augmentations, such as periodic roll that simulates point re-entries, and supervising occluded points. We demonstrate that recurrent transformers can be substantially improved for point tracking and set a new state-of-the-art on multiple benchmarks. Model and code can be found at https://tap-next-plus-plus.github.io.

Abstract

Point Tracking

Explore similar work

TAPNext++: What's Next for Tracking Any Point (TAP)?

Spatial Orthogonal Refinement for Robust RGB-Event Visual Object Tracking

Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking