SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking

Tomáš Čížek; Matej Straka; Illia Volkov; Matěj Kripner; Ervin Macić; Jiri Matas; Klara Janouskova; Martin Schmid

cs.CVOct 14, 2026

SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking

Authors: Tomáš Čížek, Matej Straka, Illia Volkov, Matěj Kripner, Ervin Macić, Jiri Matas, Klara Janouskova, Martin Schmid

Organizations: Charles University, Prague · Czech Technical University in Prague · University of Oxford · EquiLibre Technologies, Inc.

Abstract

Segment Anything Model 3 (SAM 3) is a foundation model for image and video segmentation that sets the state of the art in visual object tracking. Its tracker, inherited from SAM 2, stores information from past frames in a memory bank, managed by a simple First-In-First-Out rule, allowing temporal consistency in complex videos. Several recent methods improve SAM 2 with hand-crafted memory update rules, each targeting a specific failure mode, such as distractors, occlusions, or object motion. However, their gains transfer only partially to the stronger SAM 3 backbone. We instead present a more general, data-driven approach that frames memory control as a sequential decision-making task. SAM3RL is an extension of the SAM 3 tracker with a lightweight memory controller, trained by reinforcement learning, that decides which frames enter the memory bank in order to maximize tracking performance. SAM3RL sets a new state of the art on 10 of 12 tracking and video object segmentation benchmarks, outperforming all hand-crafted rules without any runtime overhead. The same approach applied to SAM 2 gives SAM2RL, which outperforms hand-crafted rules on 9 of the 12 benchmarks, showing the generality of our method across model generations.

Figures & tables

Figure 1: Gains from hand-crafted rules vs. our learned memory control SAM3RL (left) and SAM2RL (right). Each axis is a tracking or video segmentation benchmark, spanning three categories: VOT(S) challenges (VOTS

{}_{\text{2025/2026}}

, VOTS-RT

{}_{\text{2025}}

, and VOT-LT

{}_{\text{2021/2022}}

), visual object tracking (DiDi, LaSOT, LaSOT

{}_{\text{ext}}

), and video object segmentation (MOSE

{}_{\text{v2}}

, SA-V

{}_{\text{val/test}}

, LVOS

{}_{\text{v2}}

). The radius shows the change in that benchmark’s standard metric relative to the corresponding baseline tracker (SAM 3 or SAM 2). The outer labels give our score on each benchmark. The compared methods modify what the tracker stores in the memory bank, and in most cases also mask selection or the number of banks. In contrast, our policy operates with the memory bank only. The compared methods are evaluated on SAM 2 as published and on SAM 3 after our porting to that backbone.

Figure 2: Learned memory control for SAM-based trackers. The frozen tracker (top) embeds the frame $x_{t}$ and conditions it through memory attention on the current memory bank $B_{t}$ . The decoder predicts the mask $\hat{y}_{t}$ , and the memory encoder fuses it with $x_{t}$ into a new memory $m_{t}$ . The bank has capacity seven, where slot $0$ is permanently reserved for the conditioning frame. Appending $m_{t}$ to slot $7$ therefore overfills the bank, so one of the slots $1,\dots,7$ must be evicted. Our controller $\pi_{\theta}$ observes the state $s_{t}$ , which collects temporal, confidence, semantic, and geometric features of all eight slots, and picks the action $a_{t}\in\{1,\dots,7\}$ that removes the corresponding memory (here $a_{t}=3$ ). The surviving slots form the updated bank $B_{t+1}$ that the tracker reads at step $t+1$ . During training, $\pi_{\theta}$ receives the reward $r_{t}$ , scoring $\hat{y}_{t}$ against the annotation $y_{t}$ , and maximizes the expected discounted return $\mathbb{E}_{\pi_{\theta}}\!\big[\sum_{k\geq 0}\gamma^{k}r_{t+k+1}\big]$ . At inference, $\pi_{\theta}$ is fixed and the actions are taken greedily.

		Quality			F-score
Model	FPS	VOTS ${}_{\text{2026}}$	VOTS ${}_{\text{2025}}$	VOTS-RT ${}_{\text{2025}}$	VOT-LT ${}_{\text{2021}}$	VOT-LT ${}_{\text{2022}}$
SAM 3	15.3	66.4	73.4	69.1	77.1	62.8
SAMURAI3	15.1	64.0 -2.4	72.2 -1.2	71.5 +2.4	77.4 +0.3	63.4 +0.6
SAM3Long	9.3	67.9 +1.5	75.2 +1.8	–	77.1 ±0.0	64.5 +1.7
DAM4SAM3	14.2	66.7 +0.3	73.3 -0.1	69.5 +0.4	77.6 +0.5	63.4 +0.6
SAMITE3	12.7	65.4 -1.0	73.6 +0.2	72.7 +3.6	78.3 +1.2	66.2 +3.4

Table 1: Performance on VOT(S) challenges. We report standard metrics for the evaluated challenges. Deltas indicate improvement or decline relative to SAM 3. Bold indicates the best.

Table 4

			Quality			F-score		AUC
Model	$s$	FPS	VOTS ${}_{\text{2026}}$	VOTS ${}_{\text{2025}}$	DiDi	VOT-LT ${}_{\text{2021}}$	VOT-LT ${}_{\text{2022}}$	LaSOT	LaSOT ${}_{\text{ext}}$
SAM 3	$1$	15.3	66.4	73.4	71.4	77.1	62.8	75.9	61.7
SAM3RL	$1$	14.9	69.3 +2.9	75.4 +2.0	71.7 +0.3	77.2 +0.1	65.6 +2.8	77.1 +1.2	65.2 +3.5
SAM 3	$6$	15.5	55.5 -10.9	66.9 -6.5	60.5 -10.9	76.2 -0.9	61.8 -1.0	72.0 -3.9	57.4 -4.3
SAM 3 ${}_{\text{ours}}$	$6$	15.8	68.1 +1.7	75.9 +2.5	71.2 -0.2	75.7 -1.4	65.6 +2.8	77.4 +1.5	64.8 +3.1
SAM3RL	$6$	16.0	70.0 +3.6	76.9 +3.5	72.0 +0.6	79.0 +1.9	68.3 +5.5	78.4 +2.5	64.7 +3.0

Table 4: Memory stride ablation. All benchmarks here run at 30 FPS, so

s=6

matches the 5 FPS training and

s=1

updates every frame. SAM 3 at

s=1

is the original tracker. SAM 3 at

s=6

is the official strided variant of SAM 3, which encodes memory and reselects the bank every frame

t

, keeping

t-1

frame plus the frames

t-1-s,t-1-2s,\dots

that pass SAM 3’s threshold. SAM3RL and SAM 3

{}_{\text{ours}}

update the bank only every

s

th frame and hold it fixed in between, with SAM 3

{}_{\text{ours}}

using SAM 3’s selection rule. Deltas are relative to SAM 3 at

s=1

. Bold indicates the best.

Figure 5: Admission rate of incoming frames by SAM3RL follows object change. An LVOS ${}_{\text{v2}}$ video sequence in which a rabbit settles and hops off twice over 60 s. Object motion is the shift of the ground-truth box center between consecutive decisions, normalized by the square root of the box area. Appearance change is the DINOv3 ( Siméoni et al., 2025 ) cosine distance between consecutive ground-truth crops. The middle panel is the admission rate in a 5 s window, one tick per admitted decision. Admissions are sparse while the rabbit stays still (gray) and dense when both curves rise.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

		Quality			F-score
Model	FPS	VOTS ${}_{\text{2026}}$	VOTS ${}_{\text{2025}}$	VOTS-RT ${}_{\text{2025}}$	VOT-LT ${}_{\text{2021}}$	VOT-LT ${}_{\text{2022}}$
SAM 3	15.3	66.4	73.4	69.1	77.1	62.8
SAMURAI3	15.1	64.0 -2.4	72.2 -1.2	71.5 +2.4	77.4 +0.3	63.4 +0.6
SAM3Long	9.3	67.9 +1.5	75.2 +1.8	–	77.1 ±0.0	64.5 +1.7
DAM4SAM3	14.2	66.7 +0.3	73.3 -0.1	69.5 +0.4	77.6 +0.5	63.4 +0.6
SAMITE3	12.7	65.4 -1.0	73.6 +0.2	72.7 +3.6	78.3 +1.2	66.2 +3.4

Appendix

Table 5: Extended performance on VOT(S) challenges. We report standard metrics for evaluated challenges. Deltas indicate improvement or decline relative to SAM 2 and SAM 3. Bold = best.

	Quality	AUC		AO
Model	DiDi	LaSOT	LaSOT ${}_{\text{ext}}$	GOT-10k
SAM 3	71.4	75.9	61.7	84.7
SAMURAI3	71.8 +0.4	76.3 +0.4	63.5 +1.8	85.0 +0.3
SAM3Long	71.7 +0.3	76.1 +0.2	62.0 +0.3	84.9 +0.2
DAM4SAM3	71.8 +0.4	77.3 +1.4	65.0 +3.3	84.8 +0.1
SAMITE3	71.8 +0.4	77.6 +1.7	64.7 +3.0	84.5 -0.2

Appendix

Table 6: Extended performance on VOT datasets. We report standard metrics for the evaluated datasets. Deltas indicate improvement or decline relative to SAM 2 and SAM 3. Bold = best.

		$\mathcal{J\&\dot{F}}$	$\mathcal{J\&F}$				$\mathcal{G}$
Model	FPS	MOSE ${}_{\text{v}2}$	SA-V ${}_{\text{val}}$	SA-V ${}_{\text{test}}$	LVOS ${}_{\text{v2}}$	DAVIS ${}_{\text{2017}}$	YT-VOS ${}_{\text{2019}}$
SAM 3	15.7	59.8	82.9	84.6	88.7	91.6	89.6
SAMURAI3	15.2	56.6 -3.2	78.7 -4.2	81.4 -3.2	88.4 -0.3	91.3 -0.3	89.2 -0.4
SAM3Long	9.4	59.3 -0.5	82.5 -0.4	85.0 +0.4	89.0 +0.3	91.3 -0.3	89.5 -0.1
DAM4SAM3	14.5	58.0 -1.8	80.4 -2.5	83.1 -1.5	88.9 +0.2	91.6 ±0.0	89.4 -0.2
SAMITE3	12.8	53.4 -6.4	81.1 -1.8	82.2 -2.4	89.3 +0.6	90.7 -0.9	88.9 -0.7

Appendix

Table 7: Extended performance on VOS datasets. We report standard metrics for the evaluated datasets. Deltas indicate improvement or decline relative to SAM 2 and SAM 3. Bold = best.

						Length (s)
Dataset	Seq.	Obj.	FPS	$s$	Frames	Avg.	Min	Max
VOTS ${}_{\text{2026}}$ ( Kristan et al., 2026 )	170	375	30	6	1860	62.0	2.1	356.6
VOTS(-RT) ${}_{\text{2025}}$ ( Kristan et al., 2023b ; Kristan et al., 2025 )	144	341	30	6	2074	69.1	2.1	356.6
VOT-LT ${}_{\text{2021}}$ ( Kristan et al., 2021 )	50	50	30	6	4306	143.5	46.3	990.0
VOT-LT ${}_{\text{2022}}$ ( Kristan et al., 2023a )	50	50	30	6	3366	112.2	32.6	356.6
DiDi ( Videnović et al., 2026 )	180	180	30	6	1527	50.9	2.1	333.3

Appendix

Table 8: Evaluation benchmarks with VOT(S) challenges (top), VOT datasets (middle), VOS datasets (bottom). Seq. and Obj. are the numbers of sequences and annotated targets, respectively. FPS is the annotation frame rate, Frames is the average number of frames per sequence, and

s

is the stride defined in Section 3.2 . The three length columns give the average, shortest, and longest sequence duration in seconds. SA-V videos are recorded at 24 FPS, but annotated at 6.

		Quality		AUC		AO
Source	Model	DiDi	VOTS ${}_{\text{2025}}$	LaSOT	LaSOT ${}_{\text{ext}}$	GOT-10k
SAMURAI ( Yang et al., 2026 )	SAM 2.1 L	–	–	68.5	58.6	–
SAMURAI ( Yang et al., 2026 )	SAMURAI	–	–	74.2	61.0	81.7
SAM2Long ( Ding et al., 2025b )	SAM 2.1 L	–	–	70.0	56.9	80.7
SAM2Long ( Ding et al., 2025b )	SAM2Long	–	–	73.9	60.9	81.1
SAMITE ( Xu et al., 2025 )	SAM 2.1 L	–	–	68.5	58.6	–

Appendix

Table 9: Performance on tracking benchmarks across previous work. For each work, we list the numbers for baselines as reported there (whether independently measured, or cited from prior work). The last block reports our own measurements. Hyphen means the benchmark was not reported.

		$\mathcal{J\&\dot{F}}$	$\mathcal{J\&F}$				$\mathcal{G}$
Source	Model	MOSE ${}_{\text{v2}}$	SA-V ${}_{\text{val}}$	SA-V ${}_{\text{test}}$	LVOS ${}_{\text{v2}}$	DAVIS ${}_{\text{2017}}$	YT-VOS ${}_{\text{2019}}$
MOSE ${}_{\text{v}2}$ ( Ding et al., 2025a )	SAM 2.1 L	49.4	–	78.4	84.0	90.7	89.3
	SAMURAI	51.1	–	–	–	–	–
	DAM4SAM	51.2	–	–	–	–	–
	SAM2Long	51.5	–	81.2	85.3	88.8	90.2
DAM4SAM ( Videnović et al., 2026 )	SAM 2.1 L	–	–	–	80.6	–	–

Appendix

Table 10: Performance on VOS datasets across previous work. For each work, we list the numbers for baselines as reported there (whether independently measured, or cited from prior work). The last block presents our own measurements. Hyphen means the benchmark was not reported.

Table 14

Hyperparameter	Value
Task
Backbones	SAM 3, frozen; image size $1008\times 1008$ , bf16
	SAM 2.1 L, frozen; image size $1024\times 1024$ , bf16
Memory bank size	7 slots
Action space	7 (evict one slot after appending the current frame)
Trainable parameters	6.45 M (actor 3.23 M, critic 3.22 M)

Appendix

Table 13: Training hyperparameters for SAM2RL and SAM3RL. Both backbones are frozen. Only the actor–critic is optimized on pre-extracted image embeddings.

	Quality	F-score	AUC		$\mathcal{J}\&\mathcal{F}$
Actor input	VOTS ${}_{\text{2026}}$	VOT-LT ${}_{\text{2022}}$	LaSOT	LaSOT ${}_{\text{ext}}$	MOSE ${}_{\text{v2}}$
SAM3RL (all inputs)	69.5 $\pm$ 0.2	68.5 $\pm$ 0.3	78.0 $\pm$ 0.3	64.6 $\pm$ 0.1	60.6 $\pm$ 0.3
w/o temporal	69.1 $\pm$ 0.3	68.9 $\pm$ 0.5	78.0 $\pm$ 0.1	64.3 $\pm$ 0.3	60.7 $\pm$ 0.3
w/o confidence	69.2 $\pm$ 0.2	68.4 $\pm$ 0.4	77.9 $\pm$ 0.1	64.1 $\pm$ 0.1	60.4 $\pm$ 0.1
w/o semantic	69.1 $\pm$ 0.4	67.3 $\pm$ 0.4	78.0 $\pm$ 0.2	64.5 $\pm$ 0.2	60.2 $\pm$ 0.2
w/o geometry	69.2 $\pm$ 0.2	68.2 $\pm$ 0.2	77.8 $\pm$ 0.1	64.5 $\pm$ 0.2	59.7 $\pm$ 0.2

Appendix

Table 14: Delete-one actor input feature group ablation. Each row removes one feature group from the actor input. Everything else, including the critic input, is unchanged. Every entry is the mean over three seeds, with the standard error of the mean in subscript. Cell color encodes the difference to the full-input SAM3RL mean, green for a gain and red for a drop, darker for a larger difference. Bold indicates the best.

	Quality	F-score	AUC		$\mathcal{J}\&\mathcal{F}$
Actor input	VOTS ${}_{\text{2026}}$	VOT-LT ${}_{\text{2022}}$	LaSOT	LaSOT ${}_{\text{ext}}$	MOSE ${}_{\text{v2}}$
SAM3RL (all inputs)	69.5 $\pm$ 0.2	68.5 $\pm$ 0.3	78.0 $\pm$ 0.3	64.6 $\pm$ 0.1	60.6 $\pm$ 0.3
only temporal	68.7 $\pm$ 0.3	68.3 $\pm$ 0.0	77.6 $\pm$ 0.2	64.1 $\pm$ 0.1	59.2 $\pm$ 0.1
only confidence	68.7 $\pm$ 0.2	66.3 $\pm$ 0.2	77.7 $\pm$ 0.2	64.8 $\pm$ 0.1	60.1 $\pm$ 0.6
only semantic	69.1 $\pm$ 0.4	68.8 $\pm$ 0.3	78.1 $\pm$ 0.3	63.7 $\pm$ 0.4	59.8 $\pm$ 0.2
only geometry	68.8 $\pm$ 0.2	67.6 $\pm$ 0.6	77.7 $\pm$ 0.2	63.9 $\pm$ 0.2	58.9 $\pm$ 0.8

Appendix

Table 15: Keep-one actor input feature group ablation. Each row keeps one feature group in the actor input and removes the other three. Everything else, including the critic input, is unchanged. Every entry is the mean over three seeds, with the standard error of the mean in subscript. Cell color encodes the difference to the full-input SAM3RL mean, green for a gain and red for a drop, darker for a larger difference. Bold indicates the best.

Figure 10: Admission rate of incoming frames by SAM3RL follows object change II. A 75 s DiDi video sequence in which a car cruises steadily in one lane and then starts moving quickly, changing lanes until the end. Object motion is the shift of the ground-truth box center between consecutive decisions, normalized by the square root of the box area. Appearance change is the DINOv3 ( Siméoni et al., 2025 ) cosine distance between consecutive ground-truth crops. The middle panel is the admission rate in a 5 s window, one tick per admitted decision. Admissions are sparse during the cruise and dense in the lane-changing interval (gray) when object motion rises.

Figure 11: SAM3RL holds diverse pre-absence frames through a long absence. A VOT-LT 2021 sequence captured from the onboard camera of a formula car, in which the car ahead leaves the field of view for about 22 s. The top row shows four selected frames (a–d) with the ground truth and the prediction of each method. The middle panel shows the per-frame tracking quality of each method in the selected time window, with the absence shaded red. The bottom panel shows the memory bank content of each method at the four selected frames, one row per selected frame and method and one tick per stored frame. The prompt frame at 0 s is on the left, and an arrow gives the number of stored frames older than the selected window. SAM 3 correctly predicts no target throughout the absence, with its bank fixed on consecutive frames just before it, in which the car is mostly out of view. It does not reacquire the car after its return. DAM4SAM3 wrongly predicts the target at several points during the absence, and by (b) its bank holds only frames from the absence. After the return, its predictions miss the car. SAM3RL also predicts no target during the absence. When the car leaves, its bank holds frames spread over the last few seconds before the absence and a few from the start of the absence, and at (b) and (c) it still holds nearly all of them. SAM3RL reacquires the car about 1 s after its return, and by (d) half of its bank holds frames from after the return, while the pre-absence frames remain.

Figure 12: SAM3RL holds onto very old frames and successfully reacquires. A LaSOT test sequence of a military ceremony, in which the tracked soldier’s hat leaves the field of view for about 5 s among other soldiers wearing near-identical hats. The top row shows four selected frames (a–d) with the ground truth and the prediction of each method. The middle panel shows the per-frame tracking quality of each method in the selected time window, with the absence shaded red. The bottom panel shows the memory bank content of each method at the four selected frames, one row per selected frame and method and one tick per stored frame. The prompt frame at 0 s is on the left, and an arrow gives the number of stored frames older than the selected window. SAM 3 correctly predicts no target throughout the absence, with its bank holding the last five frames before it, and reacquires the hat on the first frame after its return. DAM4SAM3 also predicts no target at the start of the absence, but shortly after it locks onto another soldier’s hat. By (b) it has evicted all pre-absence frames, and its bank holds only frames in which it tracks the wrong hat. It keeps following that hat for the rest of the sequence, also after the target returns. SAM3RL correctly predicts no target throughout the absence, but its bank barely changes. When the hat leaves the view, five of its six slots hold frames that are 20–30 s old. These stay in the bank through more than 30 s of tracking, from before (a) until after (d), and only the newest slot turns over. SAM3RL thus keeps the target’s appearance in memory throughout the gap and reacquires the hat on the first frame after its return.

Figure 13: SAM3RL holds sparse old frames and reacquires late. A VOT-LT 2021 sequence capturing a longboarder from behind, who goes out of view on the left for about 3 s. The top row shows four selected frames (a–d) with the ground truth and the prediction of each method. The middle panel shows the per-frame tracking quality of each method in the selected time window, with the absence shaded red. The bottom panel shows the memory bank content of each method at the four selected frames, one row per selected frame and method and one tick per stored frame. The prompt frame at 0 s is on the left, and an arrow gives the number of stored frames older than the selected window. SAM 3 correctly reports the target absent on most frames of the gap and reacquires successfully. DAM4SAM3 keeps 3 old frames and 3 recent ones in all selected frames, although during the absence the 3 recent ones are the pre-absence ones. SAM3RL carries four frames from before the absence through the gap, which are 4–15 s old at the start of the absence and all show the rider small and near the center of the frame, while the rider comes back large and on the left side. The memory bank frames that would match that view all passed through the two newest slots and are not present in the bank of SAM3RL at the end of the absence, but are present in the banks of SAM 3 and DAM4SAM3. SAM3RL fails to reacquire for 4 s after the return, while SAM 3 and DAM4SAM3 reacquire immediately. Once the rider moves back towards the center, SAM3RL reacquires and updates its memory bank.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 175 papers

Jun 23, 2026

SENTRY: SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking

We revisit the memory update mechanism in SAM2-based visual object tracking and identify confidence-only mask selection as the dominant cause of drift under occlusion, rapid motion, and distractors. We introduce SENTRY, a training-free, plug-and-play, refine-before-write module that validates each memory update for short-horizon temporal consistency before committing it. SENTRY aggregates diverse segmentation hypotheses per frame, backtracks them into short tracklets, and uses neighbor-aware cycle-consistent matching against recent trajectories to favor temporally and geometrically consistent masks. It leaves the base architecture untouched, replacing confidence-driven writes with consistency-validated ones. For fair evaluation, we re-evaluate major open-source SAM2-based trackers across all available scales and datasets, filling gaps in prior reports. Integrated into five strong baselines, SENTRY delivers consistent gains across nine benchmarks, achieving new zero-shot SOTA on LaSOT, LaSOT_ext, GOT-10k, VOT20, VOT22, and DiDi. Despite these checks, the SAM2-L version runs at 32.8 FPS on an A100, and across compatible hosts adds only about 0.4--0.6 GB VRAM. Our results provide the first unified all-scale evaluation of SAM2-based trackers and show that enforcing temporal validity at write time stabilizes memory-augmented tracking without retraining. Project page: https://hamadya.github.io/SENTRY/page/

Mohamad Alansari, Yonathan Michael, Hasan AlMarzouqi +3

Khalifa University, Abu Dhabi, UAE · University of Western Australia, Australia

Memory-Augmented Neural Networks Visual Object Tracking Segment Anything Models

Ask PDF

Aug 23, 2026

SAM3Dual: A 3rd Place Solution to the MOSEv2 Track, 8th LSVOS Challenge

We present SAM3Dual, our third-place solution to the MOSEv2 track of the 8th Large-scale Video Object Segmentation (LSVOS) Challenge at ECCV 2026. SAM3Dual is a training-free inference extension of pretrained SAM 3 that explicitly separates temporal memory into a short-term branch for recent observations and a long-term branch for interval-sampled historical representations. The two memory responses are combined using a deterministic sequence-relative fusion schedule and conservatively modulated by the previous-frame object confidence. All pretrained SAM 3 parameters remain frozen, requiring no task-specific training, fine-tuning, test-time training, or online parameter optimization. The complete system achieved an official J&F score of 64.37 and ranked third in the MOSEv2 track. This result highlights the potential of reorganizing temporal memory entirely at inference time to obtain competitive long-term VOS performance while preserving the pretrained model.

JeongRae Kim, Chaehyun Kim, Changwon Lim

Chung-Ang University, Seoul, Republic of Korea

Video Object Segmentation

Ask PDF

Sep 23, 2026

LiAM-SAM: Lifecycle-Aware Memory for Robust SAM2-Based MOT

Segmentation-based multi-object tracking (MOT) with foundation video models such as SAM2 offers strong localization quality, yet remains fragile in crowded, real-world scenes. In detector-prompted SAM2 pipelines, failures typically arise at three stages of the object lifecycle: (i) erroneous or duplicate track initiation, (ii) memory drift during close interactions, and (iii) unreliable re-identification after long occlusions or re-entry. These errors corrupt object memory and accumulate over time, making long-horizon tracking unstable. In this paper, we reframe MOT as a lifecycle memory integrity problem. We present LiAM-SAM, a Lifecycle-Aware Memory (LiAM) framework with targeted mechanisms for each of the three failure modes. At track birth, to prevent faulty or duplicate initiations, we apply contrastive track initiation, which conditions each prompt on existing nearby tracked instances. To preserve memory integrity during strong interactions, we introduce motion- and geometry-grounded memory correction that resolves interaction confusions and suppresses drift. For reliable re-identification after disappearance, we maintain an adaptive context memory that promotes diverse and trustworthy references as long-term identity anchors. Finally, similarity aware spatial pruning optionally selects the memory tokens to retain at cross-attention time, improving efficiency with minimal accuracy loss. LiAM-SAM represents a modular, detector-agnostic, SAM2-based MOT system that achieves state-of-the-art HOTA and IDF1 on the evaluated benchmarks. In association-challenging environments, our ablations show that LiAM improves a detector+SAM2 baseline by +10.5 HOTA, +17.4 AssA, and reduces identity switches by 96%.

Grégoire Francisco, Alessandro D'Amico, Samuele Costantini +2

Toyota Motor Europe, Zaventem, Belgium

Visual Object Tracking Multi-Object Tracking

Ask PDF

Abstract

Visual Object Tracking

Explore similar work

SENTRY: SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking

SAM3Dual: A 3rd Place Solution to the MOSEv2 Track, 8th LSVOS Challenge

LiAM-SAM: Lifecycle-Aware Memory for Robust SAM2-Based MOT