AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation

Woojeong Jin; Jaeho Lee; Heeseong Shin; Seungho Jang; Junhwan Heo; Seungryong Kim

cs.CVMar 24, 2026

AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation

Authors: Woojeong Jin, Jaeho Lee, Heeseong Shin, Seungho Jang, Junhwan Heo, Seungryong Kim

Organizations: KAIST AI

Abstract

Referring Video Object Segmentation (RVOS) aims to segment a target object throughout a video given a natural language query. Training-free methods for this task follow a common pipeline: a MLLM selects keyframes, grounds the referred object within those frames, and a video segmentation model propagates the results. While intuitive, this design asks the MLLM to make temporal decisions before any object-level evidence is available, limiting both reasoning quality and spatio-temporal coverage. To overcome this, we propose AgentRVOS, a training-free agentic pipeline built on the complementary strengths of SAM3 and a MLLM. Given a concept derived from the query, SAM3 provides reliable perception over the full spatio-temporal extent through generated mask tracks. The MLLM then identifies the target through query-grounded reasoning over this object-level evidence, iteratively pruning guided by SAM3's temporal existence information. Extensive experiments show that AgentRVOS achieves state-of-the-art performance among training-free methods across multiple benchmarks, with consistent results across diverse MLLM backbones. Our project page is available at: https://cvlab-kaist.github.io/AgentRVOS/.

Figures & tables

Figure 1: Teaser. AgentRVOS is a training-free agentic pipeline built on the complementary strengths of SAM3 [ 4 ] and an MLLM [ 1 , 27 ] . The MLLM first uses SAM3 to generate candidate mask tracks, then iteratively prunes them through query-grounded reasoning over object-level evidence.

Figure 2: Complementary concept of SAM3 and MLLM. SAM3 [ 4 ] can precisely identify objects without missing a single frame, but struggles with complex queries. MLLMs [ 1 , 27 , 21 ] , on the other hand, offer strong reasoning capabilities, but operate on sparse frames and struggle with non-salient objects. AgentRVOS combines the advantages of both SAM3 and MLLM, by interleaving the two models in a complementary manner.

Figure 3: Overall pipeline. Given a video and a natural language query, our pipeline operates in two phases. In Candidate Mask Track Generation (Sec. 3.2 ), the MLLM first analyzes the query to extract concepts, which SAM3 uses to produce temporally consistent candidate mask tracks; this process iterates to ensure sufficient coverage. In Iterative Spatio-temporal Pruning (Sec. 3.3 ), the MLLM reasons over the candidate pool, classifying each candidate as Accepted , Rejected , or Uncertain , while progressively narrowing the spatio-temporal scope until convergence.

Method	MLLM	MeViS			ReVOS									ReasonVOS
		$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	Referring			Reasoning			Overall			$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
					$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
Training-based Methods
VideoLISA [ 3 ]	LLaVA-3.8B	41.3	47.6	44.4	-	-	-	-	-	-	-	-	-	45.1	49.9	47.5
VISA [ 38 ]	Chat-UniVi-13B	41.8	47.1	44.5	55.6	59.1	57.4	42.0	46.7	44.3	48.8	52.9	50.9	-	-	-
HyperSeg [ 34 ]	Mipha-3B	-	-	-	56.0	60.9	58.5	50.2	55.8	53.0	53.1	58.4	55.7	-	-	-

Table 1: Comparison with state-of-the-art methods on Referring and Reasoning VOS Benchmarks : MeViS [ 7 ] , ReVOS [ 38 ] and ReasonVOS [ 3 ] . Qwen3-VL-8B-T and Qwen3-VL-32B-T indicate Qwen3-VL-8B-Thinking [ 1 ] and Qwen3-VL-32B-Thinking model, respectively. The best performing results are presented in bold , while the second-best results are underlined .

\dagger

denotes our reproduced results.

Figure 4: Qualitative results. AgentRVOS effectively resolves challenging scenarios such as multi-instance ambiguity and temporal reasoning, accurately segmenting the referred objects.

Component	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
(a) w/o Retry in Concept Extraction	65.6	71.8	68.7
(b) w/o Iteration in Iterative Spatio-temporal Pruning	63.0	68.1	65.6
(c) w/o Temporal Scope Pruning	64.2	70.4	67.3
AgentRVOS (Full Pipeline)	67.1	73.6	70.3

Table 2: Component analysis. We provide ablation study for our core components– (a) retry for the concept extraction in the Candidate Mask Track Generation phase, and (b) the iterative process and (c) temporal scope pruning in the Iterative Spatio-temporal Pruning phase.

Max Iteration	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
1 (No Retry)	63.0	68.1	65.6
2	64.4	70.6	67.5
3	67.1	73.6	70.3
4	67.4	73.7	70.6

Table 3: Results for varying maximum number of iterations.

# of Frames	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
8	64.6	70.7	67.6
16	67.1	73.6	70.3
32	65.6	72.1	68.8
64	64.9	71.5	68.2

Table 4: Effect of the number of sampled frames.

Figure 5: Qualitative results of iteration in Iterative Spatio-temporal Pruning. We illustrate how our iterative spatio-temporal pruning progressively narrows the relevant temporal window and eliminates irrelevant track candidates. Across iterations, the remaining candidates become fewer but more query-consistent, leading to the final selected track set.

Figure 6: Qualitative results of iteration in Concept Extraction. We visualize how our iterative concept extraction progressively refines object concepts from the query. From top to bottom, the iterative process refines concepts through three distinct patterns: (a) the extracted concept class changes entirely to better match the query, (b) a specific concept is broadened to a more general one, and (c) a vague concept is narrowed to a more precise one, each converging to concepts better aligned with the query’s intent.

Iterations	Empty Mask Ratio (%)
CoT-RVS [ 16 ]	12.0
1	4.9
2	4.1 ( -0.8 )
3	3.8 ( -1.1 )

Table 5: Effects of iterations on the ratio of empty mask prediction (%).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	Empty Mask Ratio (%)	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
SAM3 [ 4 ]	38.9	36.8	44.7	40.8
AgentRVOS	3.8 ( - 35.1 )	67.1 ( + 30.3 )	73.6 ( + 28.9 )	70.3 ( + 29.5 )

Appendix

Table A1 : Ablation on MLLM reasoning. Replacing our MLLM-based reasoning pipeline with direct input of the referring expression into SAM3 leads to a substantial drop in segmentation accuracy and a sharp increase in empty mask ratio, demonstrating that MLLM reasoning is essential for handling the complexity of referring expressions.

Figure A1 : Concept extraction for referring and reasoning queries. For referring queries (a), the target is identifiable from the expression alone, so SAM3 is applied directly using the extracted concept. For reasoning queries (b), resolving the referent requires visual evidence, so the video is provided alongside the expression. Notably, SAM3 is applied across all frames rather than a sampled subset, enabling reliable detection even when the target object appears in only a small fraction of the video, as illustrated by the “ hand ” visible in just two frames.

Figure A2 : Details of appearance tool. Between candidate mask track generation and iterative spatio-temporal pruning, the appearance tool can be optionally invoked to obtain appearance information that may be obscured by visual prompting. When the agent determines that appearance-level evidence is needed for reasoning, it generates a brief phrase describing each candidate object’s appearance.

Component	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
w/o Appearance Tool	64.6	70.9	67.7
AgentRVOS	67.1	73.6	70.3

Appendix

Table A2 : Ablation for appearance tool call.

Figure A3 : Visualization of the overall pipeline of AgentRVOS. This example illustrates a case where candidate pruning is applied. AgentRVOS progressively rejects irrelevant objects across iterations and identifies the correct target through proper iterative reasoning.

Figure A4 : Visualization of the overall pipeline of AgentRVOS. This example illustrates a case requiring both candidate pruning and temporal scope pruning . After spatially narrowing the candidates, the pipeline further restricts the temporal scope to resolve the remaining ambiguity.

Figure A5 : Qualitative results in MeViS. MeViS [ 7 ] expressions require understanding object motion and spatial relationships. AgentRVOS successfully identifies targets based on motion descriptions such as walking direction and relative displacement.

Figure A6 : Qualitative results in MeViS. These examples involve distinguishing among multiple similar objects by their motion patterns, such as remaining stationary or performing a specific action first.

Figure A7 : Qualitative results in ReVOS. ReVOS [ 38 ] expressions often involve complex reasoning about object interactions and scene context. AgentRVOS correctly segments targets described through causal relationships, relative comparisons, and challenging visual conditions.

Figure A8 : Qualitative results in ReasonVOS. ReasonVOS [ 3 ] requires commonsense and world knowledge beyond direct visual cues. AgentRVOS handles expressions involving functional reasoning, hypothetical scenarios, and temporal event understanding.

Figure A9 : Prompts used in AgentRVOS (

\texttt{prompt}_{referring}

). The prompt for concept extraction, where the model receives a referring expression and extracts a core concept and a broader concept to guide SAM3 mask track generation.

Figure A10 : Prompts used in AgentRVOS (

\texttt{prompt}_{reasoning}

). The prompt for concept extraction, where the model receives the referring expression and extracts a core concept and a broader concept to guide SAM3 mask track generation.

Figure A11 : Prompts used in AgentRVOS (

\texttt{prompt}_{appearance\_requirement}

and

\texttt{prompt}_{appearance\_retrieval}

). The prompt for Appearance Tool Verification (top) determines whether the referring expression requires appearance-level evidence, and the prompt for Appearance Tool (bottom) extracts concise color and appearance descriptions from each candidate.

Figure A12 : Prompts used in AgentRVOS (

\texttt{prompt}_{select}

). The prompt for Iterative Spatio-temporal Pruning, where the model receives a candidate mask track and a language query and classifies each candidate as Accept , Reject , or Uncertain , progressively narrowing the candidate pool.

Referring Video Object Segmentation

Referring video object segmentation grounds language descriptions in video frames and produces temporally consistent masks for the described objects.

See all 42 papers

ReflexTrack: A Feedback-Driven Agent for Training-Free Referring Video Object Segmentation
Jul 27, 2026Yuanjia Li, Tianyang Xu, Tao Zhou +3Video Object Segmentation Referring Video Object Segmentation
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, GU2 7XH, UK
VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation
Mar 28, 2026Jihwan Hong, Jaeyoung DoVideo Object Segmentation Referring Video Object Segmentation
AIDAS Laboratory, 1IPAI, Seoul National University · AIDAS Laboratory, 2ECE, Seoul National University
VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation
Jun 5, 2026Ming Dai, Sen Yang, Boqiang Duan +4Video Object Segmentation Referring Video Object Segmentation
School of Automation, Southeast University, Nanjing, China · Baidu Inc, Beijing, China · College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, China

Jul 27, 2026J/K move · Enter open · S save

ReflexTrack: A Feedback-Driven Agent for Training-Free Referring Video Object Segmentation

Yuanjia Li, Tianyang Xu, Tao Zhou, Zhangyong Tang+2

School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, GU2 7XH, UK

Video Object Segmentation Referring Video Object Segmentation Iterative Refinement

Open paper AskPDF arXiv

Referring video object segmentation (RVOS) requires segmenting a target specified by natural language throughout a video. Recent agentic approaches combine multimodal large language models with promptable segmentation models to perform RVOS without task-specific training. However, most pipelines rely on one-shot spatial grounding followed by mask propagation, leaving both the initial prompts and temporal predictions largely unverified. We introduce ReflexTrack, a training-free, feedback-driven agent that closes this loop at both spatial and temporal levels. Mask-guided Spatial Refinement evaluates the mask induced by the current keyframe prompt and iteratively updates the bounding box together with positive and negative points, yielding a more reliable initialization. Video-level Mask Reflection assesses the complete mask sequence, localizes unreliable intervals, selects complementary repair keyframes, and generates candidate predictions through mask-guided re-propagation. Only candidates that provide a verified improvement are used to update the affected intervals, preserving reliable predictions elsewhere. All components remain frozen during inference. ReflexTrack achieves an overall

\mathcal{Q}

score of

69.7

on Ref-VPS and a

\mathcal{J}\&\mathcal{F}

score of

67.2

on ReasonVOS. These results demonstrate that prediction-level feedback substantially improves the reliability of training-free RVOS.

Abstract

Referring Video Object Segmentation

Explore similar work

ReflexTrack: A Feedback-Driven Agent for Training-Free Referring Video Object Segmentation

VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation

VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation