Temporally Consistent Long-Term Memory for 3D Single Object Tracking

Jaejoon Yoo; SuBeen Lee; Yerim Jeon; Miso Lee; Jae-Pil Heo

cs.CVApr 15, 2026

Temporally Consistent Long-Term Memory for 3D Single Object Tracking

Authors: Jaejoon Yoo, SuBeen Lee, Yerim Jeon, Miso Lee, Jae-Pil Heo

Organizations: Sungkyunkwan University

Abstract

3D Single Object Tracking (3D-SOT) aims to localize a target object across a sequence of LiDAR point clouds, given its 3D bounding box in the first frame. Recent methods have adopted a memory-based approach to utilize previously observed features of the target object, but remain limited to only a few recent frames. This work reveals that their temporal capacity is fundamentally constrained to short-term context due to severe temporal feature inconsistency and excessive memory overhead. To this end, we propose a robust long-term 3D-SOT framework, ChronoTrack, which preserves the temporal feature consistency while efficiently aggregating the diverse target features via long-term memory. Based on a compact set of learnable memory tokens, ChronoTrack leverages long-term information through two complementary objectives: a temporal consistency loss and a memory cycle consistency loss. The former enforces feature alignment across frames, alleviating temporal drift and improving the reliability of proposed long-term memory. In parallel, the latter encourages each token to encode diverse and discriminative target representations observed throughout the sequence via memory-point-memory cyclic walks. As a result, ChronoTrack achieves new state-of-the-art performance on multiple 3D-SOT benchmarks, demonstrating its effectiveness in long-term target modeling with compact memory while running at real-time speed of 42 FPS on a single RTX 4090 GPU. The code is available at https://github.com/ujaejoon/ChronoTrack

Figures & tables

Figure 1: (a) Illustration of the feature space in existing works versus ours. Different shapes and colors represent different parts of the object and different time indices, respectively. In existing methods, target features tend to drift as the target’s appearance changes, resulting in temporal inconsistency that diminishes the utility of earlier features in memory. In contrast, our method enforces temporal feature consistency across frames, mitigating drift and enabling more reliable memory readout. (b) Quantitative analysis of temporal feature consistency and its correlation with performance. We evaluate the temporal consistency by measuring the average cosine similarity between target features from each frame and those from a future frame. Compared to MBPTrack, our method maintains higher feature similarity as the temporal distance increases. This robustness allows our method to take advantage of longer context, whereas MBPTrack fails to benefit from increased temporal capacity and instead exhibits performance (Success and Precision) degradation.

Figure 2: Overall pipeline of ChronoTrack . The point features $F_{t}$ of the current point cloud $P_{t}$ are first extracted by the backbone network $E$ . These features are then fed to the Memory-based Feature Refiner (MFR) along with the long-term target memory $X^{\text{FG}}_{t-1}$ and short-term background memory $X^{\text{BG}}_{t-1}$ , injecting long-term target cues and short-term contextual information into the current features, resulting in target-aware features $Z_{t}$ . The decoder $D$ then predicts the 3D bounding box $B_{t}$ and targetness mask $M_{t}$ from these enriched features, and the memory is updated based on the predicted mask and current point features. To ensure the reliability of accumulated information in memory and foster memory diversity, two complementary losses are applied: temporal consistency loss $\mathcal{L}_{\text{TC}}$ and memory cycle consistency loss $\mathcal{L}_{\text{MCC}}$ .

Figure 3: Losses for temporal consistency and token diversity. (a) Target points from distant frames are transformed into canonical coordinates to identify approximate pairs of spatially aligned points that likely belong to the same part of the target object. A temporal consistency loss is then applied to each pair to enforce high feature similarity. (b) On the basis of these temporally aligned features, each memory token undergoes a two-step walk (memory $\rightarrow$ points $\rightarrow$ memory). The walk is optimized to simultaneously maximize the probability that each token returns to itself after a two-step walk and the likelihood of transiting through foreground points, thereby encouraging tokens to encode diverse and target-specific semantics.

Method	Car		Pedestrian		Van		Cyclist		Mean
Method	Suc.	Prec.	Suc.	Prec.	Suc.	Prec.	Suc.	Prec.	Suc.	Prec.
SC3D [ 4 ]	41.3	57.9	18.2	37.8	40.4	47.0	41.5	70.4	31.2	48.5
P2B [ 13 ]	56.2	72.8	28.7	49.6	40.8	48.4	32.1	44.7	42.4	60.0
STNet [ 7 ]	72.1	84.0	49.9	77.2	58.0	70.6	73.5	93.7	61.3	80.1
M 2 -Track [ 25 ]	65.5	80.8	61.5	88.2	53.8	70.7	73.2	93.5	62.9	83.4
CXTrack [ 20 ]	69.1	81.6	67.0	91.5	60.0	71.8	74.2	94.3	67.5	85.3

Table 1: Comparison with the SOTA methods on KITTI. “Mean” denotes the average results weighted by frame numbers. Bold and underline represent the best and second-best results, respectively.

Method	Car		Pedestrian		Truck		Trailer		Bus		Mean
Method	Suc.	Prec.	Suc.	Prec.	Suc.	Prec.	Suc.	Prec.	Suc.	Prec.	Suc.	Prec.
SC3D [ 4 ]	22.3	21.9	11.3	12.7	30.7	27.7	35.3	28.1	29.4	24.1	20.7	20.2
P2B [ 13 ]	38.8	42.2	28.4	52.2	43.0	41.5	49.0	40.1	33.0	27.4	36.5	45.1
M 2 -Track [ 25 ]	55.9	65.1	32.1	60.9	57.4	59.5	57.6	58.3	51.4	51.4	49.2	62.7
MBPTrack [ 21 ]	62.5	70.4	45.3	74.0	62.2	63.3	65.1	61.3	55.4	51.8	57.5	69.9
SCVTrack [ 23 ]	58.9	67.7	34.5	61.5	60.6	61.4	59.5	60.1	54.3	53.6	52.1	64.7

Table 2: Comparison with the SOTA methods on NuScenes.

Figure 4: Qualitative results on the KITTI dataset.

Method	Vehicle		Pedestrian		Mean
Method	Suc.	Prec.	Suc.	Prec.	Suc.	Prec.
P2B [ 13 ]	52.6	35.4	17.9	29.6	33.0	33.5
M 2 -Track [ 25 ]	61.1	61.6	32.0	67.3	44.6	63.5
CXTrack [ 20 ]	57.1	66.1	30.7	49.4	42.2	56.7
MBPTrack [ 21 ]	61.9	71.9	33.7	52.7	46.0	61.0
SCVTrack [ 23 ]	61.3	69.8	32.2	50.0	44.8	58.6

Table 3: Comparison with the SOTA methods on WOD. The models are pre-trained on KITTI and directly tested on the WOD validation set without fine-tuning.

$X^{\text{FG}}$	$\mathcal{L}_{\text{TC}}$	$\mathcal{L}_{\text{MCC}}$	Car	Pedestrian	Van	Cyclist	Mean
$\cdot$	$\cdot$	$\cdot$	74.0 / 86.2	67.9 / 93.5	63.6 / 76.1	77.0 / 94.6	70.5 / 88.6
✓	$\cdot$	$\cdot$	73.3 / 85.7	66.5 / 90.4	59.4 / 75.1	75.7 / 94.6	69.2 / 87.0
✓	✓	$\cdot$	76.1 / 87.0	67.7 / 92.8	63.6 / 77.0	75.4 / 94.5	71.3 / 88.8
✓	✓	✓	76.0 / 88.6	68.6 / 94.0	64.2 / 77.7	77.7 / 94.8	71.8 / 90.1

Table 4: Component ablation of ChronoTrack.

X^{\text{FG}}

indicates that token-level memory is used instead of point-level memory.

\mathcal{L}_{\text{TC}}

and

\mathcal{L}_{\text{MCC}}

denote the temporal consistency loss and memory cycle consistency loss, respectively. Success/Precision metrics are used for evaluation.

Method	KITTI
Method	S (29)	M (28)	L (28)	XL (27)
MBPTrack [ 21 ]	78.0 / 91.5	73.2 / 85.3	75.4 / 87.6	72.7 / 83.7
ChronoTrack	81.5 / 93.1	77.7 / 90.5	78.5 / 90.9	74.8 / 87.6
Improvement	+3.5 / +1.6	+4.5 / +5.2	+3.1 / +3.3	+2.1 / +3.9
Method	NuScenes
Method	S (927)	M (966)	L (883)	XL (884)

Table 5: Performance on various sequence lengths on KITTI and NuScenes. We divide all tracklets into four groups based on sequence lengths, using the 25th, 50th, and 75th percentiles as thresholds, and denote them as S, M, L, and XL. Parentheses indicate the number of tracklets per group.

Figure 5: The diversity of the memory token. Color represents the index of the most similar token for each point.

Figure 6: Ablation on temporal capacity of background memory. Mean performance on the KITTI is reported.

Figure 7: Comparison of GPU memory overhead as the temporal capacity of foreground memory increases.

Setting	Car	Pedestrian	Van	Cyclist	Mean
$K=16$	74.8 / 86.8	68.6 / 93.0	62.4 / 76.7	78.1 / 94.8	71.1 / 88.7
$K=32$	76.0 / 88.6	68.6 / 94.0	64.2 / 77.7	77.7 / 94.8	71.8 / 90.1
$K=64$	74.8 / 86.7	67.7 / 92.8	63.5 / 76.4	76.7 / 94.4	70.8 / 88.6

Table 6: Performance comparison with different numbers of memory tokens

K

on the KITTI dataset.

Setting		Car	Pedestrian	Van	Cyclist	Mean
$\tau_{\text{cycle}}$ $\;\!=$	0.07	74.9 / 86.9	68.7 / 93.2	59.7 / 74.4	76.2 / 94.7	70.9 / 88.7
$\tau_{\text{cycle}}$ $\;\!=$	0.1	76.0 / 88.6	68.6 / 94.0	64.2 / 77.7	77.7 / 94.8	71.8 / 90.1
$\tau_{\text{cycle}}$ $\;\!=$	1.0	73.6 / 86.2	68.2 / 92.4	63.9 / 78.2	76.5 / 94.8	70.4 / 88.4

Table 7: Performance comparison with different temperature

\tau_{\text{cycle}}

of memory cycle consistency loss on the KITTI dataset.

Setting	Car	Pedestrian	Van	Cyclist	Mean
$\tau_{\text{dist}}=10cm$	74.9 / 86.7	69.4 / 93.3	64.1 / 78.0	76.6 / 94.6	71.6 / 89.0
$\tau_{\text{dist}}=30cm$	76.0 / 88.6	68.6 / 94.0	64.2 / 77.7	77.7 / 94.8	71.8 / 90.1
$\tau_{\text{dist}}=70cm$	74.5 / 86.8	67.9 / 92.9	63.8 / 77.6	76.9 / 94.8	70.8 / 88.8

Table 8: Performance comparison with different distance thresholds

\tau_{\text{dist}}

on the KITTI dataset.

Mehod	# Params	Inference Time	FPS	Suc./Prec.
MBPTrack [ 21 ]	1.9M	15ms	67	73.4 / 84.8
M3SOT [ 11 ]	4.3M	48ms	21	75.9 / 87.4
ChronoTrack	2.9M	24ms	42	76.0 / 88.6

Table 9: Comparison of model size and inference time with state-of-the-art methods.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 175 papers

TFTrack: A Template-Free Framework for Efficient 3D Point Cloud Tracking
Sep 7, 2026Zhaofeng Hu, Sifan Zhou, Jiahao Nie +33D Point Tracking Visual Object Tracking
Stony Brook University · Carnegie Mellon University · Hangzhou Dianzi University +2
LIST3R: Long-sequence Instance-aware 3D Reconstruction
Jul 1, 2026Jing Gao, Wei Wang, Feiran Wang +1Object-Centric 3D Reconstruction 3D Reconstruction
Beijing Jiaotong University · University of Illinois Chicago
TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations
Sep 24, 2026Ayush Jain, Sreeharsha Paruchuri, Ishita Gupta +5Point Tracking 3D Point Tracking
Carnegie Mellon University · Meta

Sep 7, 2026J/K move · Enter open · S save

TFTrack: A Template-Free Framework for Efficient 3D Point Cloud Tracking

Zhaofeng Hu, Sifan Zhou, Jiahao Nie, Ziyu Zhao+2

Stony Brook University · Carnegie Mellon University · Hangzhou Dianzi University+2

3D Point Tracking Visual Object Tracking Autonomous Driving

Open paper AskPDF arXiv

LiDAR-based 3D Single Object Tracking (3D SOT) is critical for robotic perception and navigation and aims to localize dynamic objects across frames in sparse point clouds. Existing methods, rooted in the Siamese tracking paradigm from 2D vision, rely on costly dual-input designs and excessive motion modeling guided by template priors, hindering their efficiency. Our in-depth analysis reveals: (i) the template paradigm is redundant, as the previous bounding box center encodes sufficient historical context; (ii) complex motion modeling is unnecessary, as geometric alignment provides adequate motion priors. Based on the above findings, we propose the first Template-Free Tracking framework (TFTrack). The novel framework eliminates the need for template-search pairings and operates directly on the current frame guided solely by the prior bounding box center and size. We instantiate this paradigm into three variants: TFTrack-Voxel, TFTrack-Pillar, and TFTrack-Point, to explore different 3D representations under a unified framework, ensuring flexibility across sparse and dense scenes. Extensive experiments on KITTI and nuScenes benchmarks show that TFTrack is competitive with leading template-based trackers, while reducing FLOPs by approximately 50% and running at approximately 120 FPS. By simplifying overcomplicated motion-centric designs, TFTrack establishes a new minimalist paradigm for efficient 3D point cloud tracking, paving the way for real-time and resource-efficient deployment in embedded robotic systems, such as autonomous vehicles. The code is available at https://github.com/tftrack-anonymous/TFTrack/tree/main.

Abstract

Visual Object Tracking

Explore similar work

TFTrack: A Template-Free Framework for Efficient 3D Point Cloud Tracking

LIST3R: Long-sequence Instance-aware 3D Reconstruction

TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations