Training-Free Spatio-temporal Decoupled Reasoning Video Segmentation with Adaptive Object Memory

Zhengtong Zhu; Jiaqing Fan; Zhixuan Liu; Fanzhang Li

cs.CVMar 2, 2026

Training-Free Spatio-temporal Decoupled Reasoning Video Segmentation with Adaptive Object Memory

Authors: Zhengtong Zhu, Jiaqing Fan, Zhixuan Liu, Fanzhang Li

Organizations: School of Computer Science & Technology, Soochow University, Suzhou, China

Abstract

Reasoning Video Object Segmentation (ReasonVOS) is a challenging task that requires stable object segmentation across video sequences using implicit and complex textual inputs. Previous methods fine-tune Multimodal Large Language Models (MLLMs) to produce segmentation outputs, which demand substantial resources. Additionally, some existing methods are coupled in the processing of spatio-temporal information, which affects the temporal stability of the model to some extent. To address these issues, we propose Training-Free Spatio-temporal Decoupled Reasoning Video Segmentation with Adaptive Object Memory (SDAM). We aim to design a training-free reasoning video segmentation framework that outperforms existing methods requiring fine-tuning, using only pre-trained models. Meanwhile, we propose an Adaptive Object Memory module that selects and memorizes key objects based on motion cues in different video sequences. Finally, we propose Spatio-temporal Decoupling for stable temporal propagation. In the spatial domain, we achieve precise localization and segmentation of target objects, while in the temporal domain, we leverage key object temporal information to drive stable cross-frame propagation. Our method achieves excellent results on five benchmark datasets, including Ref-YouTubeVOS, Ref-DAVIS17, MeViS, ReasonVOS, and ReVOS.

Figures & tables

Figure 1: Overview of SDAM. (a) Some existing methods rely on image-level video understanding, establishing correspondence between text descriptions and video frames, while neglecting the inherent temporal information in video tasks. (b) Our training-free framework adaptively memorizes key object information based on motion cues in the video and the frame-level confidence jointly obtained from MLLM and SAM. Additionally, we decouple the spatio-temporal information in the video to enhance the temporal stability of the architecture during the segmentation process.

Figure 2: The overall pipeline of SDAM. Our method consists of two parts: (a) Spatio-temporal Decoupling (SD). We pass the keyframe candidates into MLLM and SAM to obtain objects information in the spatial domain, and then use the Object Tracker to propagate the key object across the temporal domain. (b) Adaptive Object Memory (AOM). We first use Motion Driven Sampler to adaptively sample keyframe candidates based on motion cues, then use Joint Keyframe Selection to select the frame with the highest confidence as the keyframe, and store the key object memory in the Object Memory Bank.

Figure 3: Motion Driven Sampler. To obtain a keyframe candidate set with richer spatio-temporal information, we propose the MDS module, which can adaptively select frames with more significant scene changes as candidates based on the motion cues in the video sequence.

Method	Publication	Ref-YouTube-VOS			Ref-DAVIS17			MeViS
Method	Publication	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&F$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&F$	$\mathcal{J}$	$\mathcal{F}$
LBDT ( lbdt )	CVPR 2022	49.4	48.2	50.6	54.5	-	-	29.3	27.8	30.8
ReferFormer ( referformer )	CVPR 2022	62.9	61.3	64.6	61.1	58.1	64.1	31.0	29.8	32.2
VLT+TC ( vlt )	TPAMI 2023	62.7	-	-	60.3	-	-	35.6	33.6	37.3
HTML ( html )	ICCV 2023	63.4	61.5	65.2	62.1	59.2	65.1	-	-	-
OnlineRefer ( online_refer )	ICCV 2023	63.5	61.6	65.5	64.8	61.6	67.7	-	-	-

Table 1: Performance comparison with previous methods on the validation sets of RefVOS datasets. The highest result in each column will be marked in bold , and the second-highest result will be underlined .

Method	Publication	Referring			Reasoning			Overall
Method	Publication	$\mathcal{J}\&F$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&F$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&F$	$\mathcal{J}$	$\mathcal{F}$
Traditional methods without reasoning ability
ReferFormer ( referformer )	CVPR 2022	32.7	31.2	34.3	23.4	21.3	25.6	28.1	26.2	29.9
LMPM ( mevis )	ICCV2023	34.1	29.0	39.1	18.8	13.3	24.3	26.4	21.2	31.7
LLM-based methods with reasoning ability
TrackGPT-7B ( track_gpt )	arXiv 2023	48.2	46.7	49.7	39.0	36.8	41.2	43.6	41.8	45.5

Table 2: Performance comparison with previous methods on the validation set of ReVOS dataset.

Traditional methods without reasoning ability
Method	Publication	ReasonVOS
Method	Publication	$\mathcal{J}\&F$	$\mathcal{J}$	$\mathcal{F}$
SOC ( soc )	NeurIPS’23	35.9	33.3	38.5
SgMg ( sgmg )	ICCV’23	36.2	33.7	38.7
OnlineRefer ( online_refer )	ICCV’23	38.7	34.6	42.9
LLM-based methods with reasoning ability

Table 3: Performance comparison with previous methods on the ResonVOS dataset.

Sampling Strategy	Ref-DAVIS17			$\text{Val}_{u}$
Sampling Strategy	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
First Frame	69.2	66.3	72.1	50.6
Global	73.4	70.8	75.9	54.4
Motion Driven Sampler	76.0	73.2	78.8	55.8

Table 4: Ablation study on sampling strategies.

Confidence Weight		Ref-DAVIS17			ReasonVOS
$a$	$1-a$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
0.4	0.6	73.9	71.4	76.4	54.1
0.5	0.5	74.7	72.3	77.1	54.8
0.6	0.4	75.8	73.1	78.5	55.1
0.75	0.25	76.0	73.2	78.8	54.7

Table 5: Ablation study on confidence weight.

Figure 4: Visualization of temporal stability analysis. We analyze the temporal stability of SDAM, MUTR, and ReferFormer on the Ref-YouTube-VOS dataset.

Number of Keyframes	Ref-DAVIS17
Number of Keyframes	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$
$n=3$	68.6	66.3	70.9
$n=2$	70.7	69.4	72.0
$n=1$	73.4	70.8	75.9

Table 6: Ablation study on the number of keyframes.

Figure 5: Qualitative results on the RefVOS (a) and ReasonVOS (b) datasets. The time steps are directed from left to right. Zoom in for better viewing.

Video Object Segmentation

Video object segmentation assigns pixel-level masks to objects across frames while preserving their identities through motion, occlusion, and change.

See all 110 papers

RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation
May 8, 2026Junwei Wen, Deshui Miao, Guangming Lu +2Video Object Segmentation Video Reasoning
Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · Pazhou Lab (Huangpu)
VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation
Jun 5, 2026Ming Dai, Sen Yang, Boqiang Duan +4Video Object Segmentation Referring Video Object Segmentation
School of Automation, Southeast University, Nanjing, China · Baidu Inc, Beijing, China · College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, China
SteerSeg: Attention Steering for Reasoning Video Segmentation
May 14, 2026Ali Cheraghian, Hamidreza Dastmalchi, Abdelwahed Khamis +3Video Object Segmentation Referring Video Object Segmentation
1Macquarie University · 2York University · 3CSIRO Data61 +1

May 8, 2026J/K move · Enter open · S save

RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation

Junwei Wen, Deshui Miao, Guangming Lu, Xin Li+1

Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · Pazhou Lab (Huangpu)

Video Object Segmentation Video Reasoning Video Frame Selection Reasoning Segmentation RL for VLMs

Open paper AskPDF arXiv

Video Reasoning Segmentation (VRS) aims to segment target objects in videos based on implicit instructions that convey human intent and temporal logic. Existing MLLM-based methods predict masks with a [SEG] token after selecting frames via simple sampling or an auxiliary MLLM, where limited supervision and frame-language similarity rules often yield narrow-scope keyframe choices that weaken holistic temporal understanding and lead to brittle localization in complex multi-object scenes. To address these issues, we introduce RCoT-Seg, a video-of-thought framework that factorizes VRS into temporal video reasoning (TVR) and keyframe target perception (KTP), explicitly separating temporal reasoning from spatial perception. Specifically, in the TVR stage, an agentic keyframe selection module, initialized with a curated CoT-start corpus and refined by GRPO under task-aligned rewards, is proposed to generate and reselect the keyframe through self-evaluation, strengthening moment localization and temporal reasoning. In the KTP stage, RCoT-Seg performs high-resolution segmentation on the selected frame and propagates masks with SAM2-based methods across the sequence, replacing heuristic sampling and external selectors while improving spatial precision and inter-frame consistency. Extensive experimental results demonstrate that the proposed RCoT-Seg achieves favorable performance against the state-of-the-art methods. The code and models will be publicly released at https://github.com/Victor-wjw/RCoT-Seg.

Abstract

Video Object Segmentation

Explore similar work

RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation

VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation

SteerSeg: Attention Steering for Reasoning Video Segmentation