cs.CVOct 14, 2026

SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking

Authors: Tomáš Čížek, Matej Straka, Illia Volkov, Matěj Kripner, Ervin Macić, Jiri Matas, Klara Janouskova, Martin Schmid

Organizations: Charles University, Prague · Czech Technical University in Prague · University of Oxford · EquiLibre Technologies, Inc.

Abstract

Segment Anything Model 3 (SAM 3) is a foundation model for image and video segmentation that sets the state of the art in visual object tracking. Its tracker, inherited from SAM 2, stores information from past frames in a memory bank, managed by a simple First-In-First-Out rule, allowing temporal consistency in complex videos. Several recent methods improve SAM 2 with hand-crafted memory update rules, each targeting a specific failure mode, such as distractors, occlusions, or object motion. However, their gains transfer only partially to the stronger SAM 3 backbone. We instead present a more general, data-driven approach that frames memory control as a sequential decision-making task. SAM3RL is an extension of the SAM 3 tracker with a lightweight memory controller, trained by reinforcement learning, that decides which frames enter the memory bank in order to maximize tracking performance. SAM3RL sets a new state of the art on 10 of 12 tracking and video object segmentation benchmarks, outperforming all hand-crafted rules without any runtime overhead. The same approach applied to SAM 2 gives SAM2RL, which outperforms hand-crafted rules on 9 of the 12 benchmarks, showing the generality of our method across model generations.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SENTRY: SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking

    Jun 23, 2026Mohamad Alansari, Yonathan Michael, Hasan AlMarzouqi +3Memory-Augmented Neural NetworksVisual Object Tracking

  2. SAM3Dual: A 3rd Place Solution to the MOSEv2 Track, 8th LSVOS Challenge

    Aug 23, 2026JeongRae Kim, Chaehyun Kim, Changwon LimVideo Object Segmentation

  3. LiAM-SAM: Lifecycle-Aware Memory for Robust SAM2-Based MOT

    Sep 23, 2026Grégoire Francisco, Alessandro D'Amico, Samuele Costantini +2Visual Object TrackingMulti-Object Tracking