When Trackers Fail: VLM-Guided Verification and Recovery for Robust Video Object Segmentation

Valay Bundele; Susmit Agrawal; Mehran Hosseinzadeh; The Nam Nguyen; Hendrik P. A. Lensch

cs.CVOct 11, 2026

When Trackers Fail: VLM-Guided Verification and Recovery for Robust Video Object Segmentation

Authors: Valay Bundele, Susmit Agrawal, Mehran Hosseinzadeh, The Nam Nguyen, Hendrik P. A. Lensch

Organizations: University of Tübingen, Germany · Woven by Toyota, Japan

Abstract

Recent trackers built on foundation models such as the Segment Anything Model (SAM) achieve strong performance by propagating object masks through memory. However, this strength is also a key source of failure: under occlusion, reappearance, or distractors, an incorrect prediction can enter the tracking loop and be propagated over time, resulting in persistent identity drift. Existing approaches seek to prevent such failures through improved memory and temporal modeling, but once a failure occurs, the tracker has no explicit mechanism to recognize or correct it. We address this gap with Dual-Agent, a failure-aware tracking framework that complements propagation with explicit verification and recovery. We address this gap with Dual-Agent, a failure-aware tracking framework that complements propagation with explicit verification and recovery. A vision-language verification agent first reasons over spatio-temporal context to determine whether the tracker is still following the target. When a failure is detected, a correction agent uses temporal context to recover the target before tracking continues. The correction agent is trained on real SAM3 failures, augmented with targeted identity-switch and distractor perturbations that expose it to challenging ambiguities during recovery. On LVOS-v2 and SecVOS, Dual-Agent improves SAM3 by 1.2 and 2.7 $J\&F$ , respectively. Importantly, the proposed framework transfers without retraining to SAM2.1, SeC, and Cutie, as well as to long-term video object tracking benchmarks including LaSOT-ext, TNL2K, and VOT-LT2022. Our results show that explicitly reasoning about when tracking fails and how to recover provides a complementary path to robust tracking beyond improving propagation alone. Project page: valaybundele.github.io/Dual-Agent.

Figures & tables

Figure 1: Our proposed Dual-Agent framework recovers tracking errors made by the tracker. Rows show ground truth (top), the SAM3 baseline (middle), and our Dual-Agent framework (bottom).

Figure 2: Dual-Agent adds a verification and recovery layer above frozen SAM3. Shared evidence contains exemplars, tracker history and a query. Sparse triggers and confidence gating invoke verification; an incorrect decision activates the failure-trained corrector. Intermediate box proposals may be rechecked for up to three correction rounds. A separate commit gate uses box IoU below 0.5 when both boxes exist. Committed prompts update SAM3; otherwise tracking continues.

SAM3 IoU regime	Samples	SAM3 IoU	Corrector IoU	$\Delta$	Wins / Losses
$<0.1$ (lost target)	347	0.024	0.612	+0.587	270 / 11
$0.1$ – $0.4$ (severe drift)	808	0.224	0.727	+0.503	688 / 76
$0.4$ – $0.7$ (drift)	182	0.492	0.734	+0.242	120 / 16
$\geq 0.7$ (already good)	551	0.955	0.912	$-0.043$	9 / 40

Table 1: Corrector-only validation on the held-out corrector-SFT split. Each sample is one query frame from the 7-frame prompt window; results report IoU against ground truth for SAM3 and the corrector on the directly measurable subset (1,888 samples).

Method	$\mathcal{J}\&\mathcal{F}$	$J_{s}$	$F_{s}$	$J_{u}$	$F_{u}$
LWL	60.6	58.0	64.3	57.2	62.9
RDE	62.2	56.7	64.1	60.8	67.2
STCN	60.6	57.2	64.0	57.5	63.8
XMem	64.5	62.6	69.1	60.6	65.6
SAM 2.1	84.1	80.7	87.4	80.6	87.7
SAM2.1Long	85.9	81.7	88.6	83.0	90.5

Table 2: LVOS-v2 validation comparison against prior methods.

\mathcal{J}\&\mathcal{F}

is the overall score;

J_{s}/F_{s}

and

J_{u}/F_{u}

denote seen- and unseen-object region/boundary scores.

(a) Direct Attribution State at commit Count SAM3 IoU Dual IoU SAM3 missed 39 0.000 0.856 SAM3 drifted 32 0.035 0.903 SAM3 was correct 17 0.961 0.026

(b) Controller Statistics Trigger Verifier Corrector Commit Outcome Overall 1360 240 88 67/19/2 Eff IoU var 713 120 53 42/9/2 Occlusion 508 105 26 19/7/0 Reappear 139 15 9 6/3/0

Table 3: Mechanism-level analysis on SecVOS: (a) direct attribution by SAM3 state at commit time, and (b) controller activity. In (b), Verifier , Corrector , and Commit count verifier calls, corrector calls, and committed corrections; Outcome reports better / worse / neutral changes relative to SAM3.

Method	No Scene Change			Single Scene Change			Multi Scene Change			Overall
	J&F	J	F	J&F	J	F	J&F	J	F	J&F
XMem	71.9	72.0	71.8	47.0	47.9	46.2	41.9	42.4	41.4	48.4
DEVA	71.6	71.6	71.5	48.5	48.4	48.6	46.4	46.0	46.8	49.7
Cutie-base	72.5	72.2	72.8	53.0	52.9	53.2	48.3	47.8	48.9	52.7
SAM2.1	79.4	79.1	79.7	58.5	58.2	58.8	52.4	52.1	52.6	58.2
SAMURAI	81.8	81.6	81.9	60.6	60.6	60.7	59.3	58.9	59.7	62.2

Table 4: Performance comparison on SecVOS-160 for different scene-complexity subsets.

Trigger setting	J&F	J	F
Occlusion	75.4	75.3	75.5
+ reappearance	75.6	75.5	75.7
+ effective-IoU variance	76.94	76.9	77.0

Table 5: SecVOS under the Dual-Agent recipe: (a) trigger design, (b) voting, (c) refinement rounds.

	Overall		No scene change		Single change		Multiple changes
Propagator	Base	+Dual-Agent	Base	+Dual-Agent	Base	+Dual-Agent	Base	+Dual-Agent
Cutie [ 5 ]	52.7	59.2	72.5	75.9	53.0	59.5	48.3	56.6
SAM2.1 [ 19 ]	58.2	65.1	79.4	83.4	58.5	61.6	52.4	61.3
SeC [ 28 ]	70.0	72.0	84.2	84.6	69.6	72.2	67.5	70.5
SAM3 [ 4 ]	74.21	76.94	87.09	88.11	74.62	76.22	69.27	73.39

Table 6: Dual-Agent on other frozen propagators, SecVOS-160 (

\mathcal{J}\&\mathcal{F}

). Nothing is retrained.

Table 9

Panel of Figure 3 — Figure 3: Qualitative recovery on two SAM3 failure modes. Top: reappearance after a long absence; Bottom: drift on the target toy train in a cluttered scene with similar distractors. For each example, rows show ground truth (top), the SAM3 baseline (middle), and our Dual-Agent framework (bottom).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Split	Sequences	Objects	Frames / annotations
LVOS-v2	valid	140	239	every frame
SecVOS	full	160	182	every frame
LaSOT-ext	full	150	150	per-frame boxes
TNL2K	test	700	700	per-frame boxes
VOT-LT2022	full	50	50	per-frame boxes

Appendix

Table 9: Benchmarks and splits used in the paper. The annotation column distinguishes segmentation masks from tracking boxes.

Sample type	Count	Share	Supervised behavior
false_negative	14,131	17.8%	emit a fresh visible-true box from context
perturbed	12,349	15.5%	recover from jittered context boxes
low_iou	12,011	15.1%	emit a refined box over a drifted SAM3 box
correct	11,618	14.6%	mirror SAM3’s already-correct prediction
both_none	10,746	13.5%	emit visible-false on absent target
synthetic_fp	4,807	6.0%	reject artificially-injected false positive

Appendix

Table 10: Sample-type distribution of the corrector-SFT corpus (79,457 samples).

SAM3 was …	Visible target	Absent target	Total
Correct	12,385 (15.6%)	14,223 (17.9%)	26,608 (33.5%)
Incorrect	43,403 (54.6%)	9,446 (11.9%)	52,849 (66.5%)
Total	55,788 (70.2%)	23,669 (29.8%)	79,457 (100%)

Appendix

Table 11: Class balance (SAM3 correct / incorrect) cross-tabulated against query-frame visibility, on all 79,457 samples. Rows are the supervised label; columns are the visibility of target at query frame.

IoU bucket	Frames	Share
$[0.00,~0.10)$	45,598	18.2%
$[0.10,~0.30)$	15,763	6.3%
$[0.30,~0.50)$	13,595	5.4%
$[0.50,~0.70)$	11,646	4.7%
$[0.70,~0.85)$	15,922	6.4%
$[0.85,~0.95)$	39,947	16.0%

Appendix

Table 12: Histogram of context-frame SAM3 IoU against ground truth across all 249,905 non-exemplar context frames in the corpus. Mean IoU 0.683, median 0.925, standard deviation 0.388.

Difficulty	Samples	Share	Definition
easy	50,948	64.1%	at least 2 context frames with IoU $>0.5$
hard	13,357	16.8%	one good context frame
noisy	3,000	3.8%	no good context frame
no_context	12,152	15.3%	no context frames

Appendix

Table 13: Per-sample difficulty distribution, classified by how many usable frames the sample has.

Sample type	N	Mean IoU	Median IoU
correct	9,531	0.924	0.966
both_none	9,723	0.784	0.888
perturbed	9,736	0.774	0.895
synthetic_fp	4,257	0.762	0.873
false_negative	9,989	0.749	0.844
paired_occluded	3,365	0.724	0.811

Appendix

Table 14: Mean context-frame SAM3-vs-ground-truth IoU stratified by sample type (excluding the exemplar frames). Lower mean IoU corresponds to noisier context the corrector must reason over.

Panel of Figure 5 — Figure 5: Further qualitative examples of our model vs. SAM3. Top: As the target object partially reappears after a long time in a largely different context, SAM3’s score for the object drops, and it fails to identify it; this triggers our recovery mechanism, and the object is re-identified by our dual agents. Middle: Upon partial appearance of the object, SAM3 fails to find the target object, due to its significant shift from its prior appearance. However, it is recovered by a further VLM-based retrieval through our designed agents. Bottom: Due to the change of the view of the scene, a different pose of the person appears in the video, resulting in the object not being found by SAM3. Our verification agent is thus triggered and detects the need for correction, with the recovery applied afterwards by the second correction agent. For each example, rows show ground truth (top), the SAM3 baseline (middle), and our Dual-Agent framework (bottom).

(a) Agent ablation Configuration SecVOS-160 SAM3 74.21 Corrector at every trigger (verifier skipped) 74.10 Dual-Agent (ours) 76.94

(b) Commit-gate ablation Configuration Gate SecVOS-160 LVOS-v2 SAM3 – 74.21 88.2 Dual-Agent off 75.90 89.3 Dual-Agent (ours) on 76.94 89.4

Appendix

Table 15: Verification and commitment (

\mathcal{J}\&\mathcal{F}

). (a) Bypassing verification returns SecVOS-160 to the frozen-propagator level. (b) Removing the commit gate keeps most of the gain on both benchmarks.

Split	Objects	Untouched	Improved	Unchanged	Regressed
Seen	202	185	7	2	8
Unseen	37	31	5	0	1

Appendix

Table 16: Object-level intervention outcomes on LVOS-v2. Improved/unchanged/regressed counts are among modified objects, not counts of individual commits.

Figure 6: Representative failures of the unconditional corrector on already-accurate SAM3 predictions (Table 1 , IoU $\geq 0.7$ ). Each row shows the full corrector input: ground-truth exemplars (green), SAM3 context (red), and the query, with the corrector’s output in yellow. (a) A target leaving the frame edge and (b) a small distant target are wrongly predicted absent. (c) The box drifts to a look-alike (the other shoe).

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 175 papers

Sep 23, 2026

LiAM-SAM: Lifecycle-Aware Memory for Robust SAM2-Based MOT

Segmentation-based multi-object tracking (MOT) with foundation video models such as SAM2 offers strong localization quality, yet remains fragile in crowded, real-world scenes. In detector-prompted SAM2 pipelines, failures typically arise at three stages of the object lifecycle: (i) erroneous or duplicate track initiation, (ii) memory drift during close interactions, and (iii) unreliable re-identification after long occlusions or re-entry. These errors corrupt object memory and accumulate over time, making long-horizon tracking unstable. In this paper, we reframe MOT as a lifecycle memory integrity problem. We present LiAM-SAM, a Lifecycle-Aware Memory (LiAM) framework with targeted mechanisms for each of the three failure modes. At track birth, to prevent faulty or duplicate initiations, we apply contrastive track initiation, which conditions each prompt on existing nearby tracked instances. To preserve memory integrity during strong interactions, we introduce motion- and geometry-grounded memory correction that resolves interaction confusions and suppresses drift. For reliable re-identification after disappearance, we maintain an adaptive context memory that promotes diverse and trustworthy references as long-term identity anchors. Finally, similarity aware spatial pruning optionally selects the memory tokens to retain at cross-attention time, improving efficiency with minimal accuracy loss. LiAM-SAM represents a modular, detector-agnostic, SAM2-based MOT system that achieves state-of-the-art HOTA and IDF1 on the evaluated benchmarks. In association-challenging environments, our ablations show that LiAM improves a detector+SAM2 baseline by +10.5 HOTA, +17.4 AssA, and reduces identity switches by 96%.

Grégoire Francisco, Alessandro D'Amico, Samuele Costantini +2

Toyota Motor Europe, Zaventem, Belgium

Visual Object Tracking Multi-Object Tracking

Ask PDF

Aug 19, 2026

SAM2Dual: Training-Free, Dual Memory for Long-Term Video Object Segmentation

Long-term video object segmentation (VOS) remains challenging due to error accumulation under extended occlusions, re-appearance, and scene changes. Although SAM2 provides strong zero-shot performance, its streaming memory can amplify drift over long horizons when recent, unreliable predictions dominate the memory state. We propose SAM2Dual, a training-free, plug-and-play inference-time enhancement that improves long-video robustness without updating model weights. SAM2Dual introduces a Dual Memory design that explicitly separates (i) short-term memory for rapid local adaptation and (ii) long-term memory built via interval-based sampling to preserve global identity cues, combined through a gated fusion strategy. In addition, we present Text-Aware Memory (TAM), which extracts a compact word-level cue from early frames and uses text embeddings to reweight memory contributions based on semantic compatibility, supporting identity preservation when visual evidence becomes weak or ambiguous. Across long-term benchmarks, SAM2Dual consistently improves stability on long videos, raising J&F from 49.33 to 50.65 on MOSEv2 and achieving consistent gains on LVOSv2.

JeongRae Kim, Changwon Lim

Department of Applied Statistics, Chung-Ang University, Seoul 06974, Republic of Korea

Video Object Segmentation Memory-Augmented Video Understanding

Ask PDF

Oct 14, 2026

SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking

Segment Anything Model 3 (SAM 3) is a foundation model for image and video segmentation that sets the state of the art in visual object tracking. Its tracker, inherited from SAM 2, stores information from past frames in a memory bank, managed by a simple First-In-First-Out rule, allowing temporal consistency in complex videos. Several recent methods improve SAM 2 with hand-crafted memory update rules, each targeting a specific failure mode, such as distractors, occlusions, or object motion. However, their gains transfer only partially to the stronger SAM 3 backbone. We instead present a more general, data-driven approach that frames memory control as a sequential decision-making task. SAM3RL is an extension of the SAM 3 tracker with a lightweight memory controller, trained by reinforcement learning, that decides which frames enter the memory bank in order to maximize tracking performance. SAM3RL sets a new state of the art on 10 of 12 tracking and video object segmentation benchmarks, outperforming all hand-crafted rules without any runtime overhead. The same approach applied to SAM 2 gives SAM2RL, which outperforms hand-crafted rules on 9 of the 12 benchmarks, showing the generality of our method across model generations.

Tomáš Čížek, Matej Straka, Illia Volkov +5

Charles University, Prague · Czech Technical University in Prague · University of Oxford +1

Visual Object Tracking Video Object Segmentation Reinforcement Learning

Ask PDF

Abstract

Visual Object Tracking

Explore similar work

LiAM-SAM: Lifecycle-Aware Memory for Robust SAM2-Based MOT

SAM2Dual: Training-Free, Dual Memory for Long-Term Video Object Segmentation

SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking