Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation

Junwei Wen; Deshui Miao; Guangming Lu; Ming-Hsuan Yang; Xin Li; Wenjie Pei

cs.CVOct 18, 2026

Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation

Authors: Junwei Wen, Deshui Miao, Guangming Lu, Ming-Hsuan Yang, Xin Li, Wenjie Pei

Organizations: Harbin Institute of Technology, Shenzhen · University of California at Merced · Pengcheng Laboratory · Pazhou Lab (Huangpu)

Abstract

Video Reasoning Segmentation (VRS) requires a model to identify an implicitly described target and predict its pixel-level mask trajectory, while resolving the spatial relations and temporal events that determine the answer. Text-only Chain-of-Thought (CoT) can explain a target choice without making the underlying visual references explicit, leaving a video reference gap between linguistic inference and the evidence distributed across frames. We introduce Temporal Chain of Video Primitives (TCoVP), a spatiotemporally grounded form of Video CoT that organizes query-relevant observations into a chronological reasoning chain. Each primitive associates an entity reference with temporal support and timestamped spatial evidence, and the chain interleaves these references with language to guide reasoning about which entity is relevant, when the queried event holds, and where its evidence appears. The selected target primitives then provide box prompts and temporal support for video mask prediction. We construct TCoVP-SFT-15k and TCoVP-RL-13k to learn this representation through supervised fine-tuning and chain-aligned reinforcement learning. Extensive experiments across five VRS benchmarks and three model backbones demonstrate the strong performance of TCoVP, highlighting the effectiveness of organizing spatial and temporal evidence within the reasoning process. All models and code will be publicly released. Project page: https://victor-wjw.github.io/TCoVP/

Figures & tables

Figure 1: Reasoning with Temporal Chain of Video Primitives. A text-only keyframe-based prediction focuses on the later hugging event and localizes the wrong athlete. TCoVP progressively records the relevant entities as timestamped video primitives, distinguishes the hand-clasp interval from the longer presence interval, and converts the selected target primitive into a mask trajectory.

Figure 2: Constructing TCoVP supervision. Kimi-K2.5 proposes language traces interleaved with temporally ordered video primitives. Chain-consistency checking and Qwen2.5-VL primitive filtering retain primitives that can be grounded in the sampled video. The filtered examples form TCoVP-SFT-15k and TCoVP-RL-13k.

Figure 3: Chain-aligned training and mask execution. The policy generates multiple serialized TCoVP completions. Structural validity, temporal alignment, spatiotemporal grounding, and target primitive accuracy rewards produce group-normalized advantages, while the reference policy provides KL regularization. The selected target primitives are passed to SAM3 for video mask propagation.

Methods	Overall			Causal			Sequential			Counterfactual			Descriptive
Methods	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
Two-Stage Baseline
ViLA ( Lin et al., 2024 ) +ReferFormer	16.37	14.84	17.89	15.92	14.33	17.52	16.30	14.94	17.67	13.63	12.35	14.91	19.61	17.78	21.45
ViLA ( Lin et al., 2024 ) +SgMg	17.07	15.25	18.89	17.34	15.43	19.26	16.30	14.57	18.04	15.31	13.96	16.67	19.18	16.96	21.40
ViLA ( Lin et al., 2024 ) +HTR	15.72	13.90	17.53	16.46	14.60	18.31	15.11	13.39	16.83	12.61	11.22	13.99	18.37	16.10	20.63
VideoChat2 ( Li et al., 2024 ) +ReferFormer	15.38	13.84	16.93	14.74	13.10	16.37	13.95	12.69	15.21	13.12	11.80	14.43	19.82	17.89	21.74

Table 1: Performance comparison on the motion-grounded video reasoning segmentation benchmark GroundMoRe . Best results are bolded , and the second best results are underlined .

Methods	MLLM	Long-RVOS			ReasonVOS			ReVOS			MeViS
Methods	MLLM	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
SFT-based Methods
VISA-7B ( Yan et al., 2024 )	Chat-UniVi-7B	-	-	-	-	-	-	46.9	44.9	49.0	43.5	40.7	46.3
VISA-13B ( Yan et al., 2024 )	Chat-UniVi-13B	-	-	-	-	-	-	50.9	48.8	52.9	44.5	41.8	47.1
VideoLISA-3.8B ( Bai et al., 2024 )	LLaVA-Phi-3-V	14.0	-	-	47.5	45.1	49.9	-	-	-	44.4	41.3	47.6
InstructSeg-3B ( Wei et al., 2025 )	Mipha-3B	-	-	-	-	-	-	54.5	52.0	56.9	-	-	-

Table 2: Performance comparison on Video Reasoning Segmentation benchmarks. Best results are bolded , and the second best results are underlined .

Model	ReVOS	ReasonVOS	Long-RVOS	GroundMoRe	MeViS	Macro avg.
Veason-R1-3B	59.9	55.2	51.8	23.5	51.2	48.3
Text-CoT with answer-VP	62.0	65.2	60.5	46.3	54.8	57.8
TCoVP-3B	65.3	71.6	65.6	53.7	57.3	62.7

Table 3: Ablation of the reasoning representation. All entries are

\mathcal{J}\&\mathcal{F}

. The macro average uses five benchmarks once each.

Model	Long-RVOS			ReasonVOS			ReVOS			GroundMoRe
Model	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
Different Training Strategies
SFT only	56.8	56.2	57.4	63.1	60.3	65.9	59.9	57.1	62.7	43.8	42.5	45.0
SFT + chain-aligned RL	65.6	64.7	66.5	71.6	68.8	74.4	65.3	62.8	67.9	53.7	52.6	54.8
Different Reward Functions
w/o $\mathcal{R}_{ta}$ and $\mathcal{R}_{stg}$	62.5	62.1	62.9	67.4	64.5	70.3	63.1	60.4	65.8	47.9	46.8	49.1

Table 4: Ablation of chain-aligned reinforcement learning and the temporal alignment and spatiotemporal grounding rewards.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Composition of the TCoVP training datasets

Figure 5: Prompt template for spatiotemporally grounded annotation generation

Figure 6: Task-specific system prompts for TCoVP. The conventional VRS prompt permits multiple target objects, whereas the GroundMoRe prompt restricts target temporal spans to the duration of the query-relevant action and requires a single target object.

Model	Long-RVOS			ReasonVOS			ReVOS			GroundMoRe
Model	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
SAM2-based Mask Propagation
Veason-R1-3B	51.8	50.8	52.7	55.2	51.8	58.5	59.9	56.9	62.8	23.5	22.0	25.0
TCoVP-3B	64.7	63.9	65.5	69.4	66.7	72.1	64.2	61.7	66.8	54.1	52.9	55.2
SAM3-based Mask Propagation
SFT only	56.8	56.2	57.4	63.1	60.3	65.9	59.9	57.1	62.7	43.8	42.5	45.0

Appendix

Table 5: Comparison of SAM2 and SAM3 mask propagation with oracle spatial prompts.

Methods	Overall			Causal			Sequential			Counterfactual			Descriptive
Methods	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
SAM2-based Mask Propagation
Full video + predicted box	39.22	37.80	40.65	40.74	39.21	42.27	36.22	34.90	37.54	35.40	34.09	36.70	43.99	42.49	45.50
Predicted timespan + predicted box	54.06	52.93	55.20	53.44	52.22	54.67	57.46	56.46	58.45	51.39	50.36	52.41	53.96	52.70	55.22
Oracle timespan + predicted box	85.60	84.18	87.03	86.83	85.30	88.37	85.85	84.53	87.17	84.66	83.35	85.96	84.70	83.19	86.20
Full video + oracle box	47.14	45.83	48.46	47.17	45.80	48.54	44.01	42.82	45.21	44.46	43.23	45.69	52.84	51.38	54.31

Appendix

Table 6: Fine-grained oracle analysis of temporal and spatial grounding on GroundMoRe.

Model	Long-RVOS		ReasonVOS		ReVOS		GroundMoRe
Model	think	answer	think	answer	think	answer	think	answer
Different Training Strategies
SFT only	59.8	54.9	59.7	59.4	63.4	60.3	57.5	52.6
SFT + chain-aligned RL	66.9	64.3	64.5	65.1	66.7	65.4	62.5	59.4
Different Reward Functions
w/o $\mathcal{R}_{ta}$ and $\mathcal{R}_{stg}$	63.6	60.8	59.9	61.6	65.1	62.9	59.0	53.8

Appendix

Table 7: Intermediate bounding-box IoU across four VRS benchmarks.

Model	Overall		Causal		Sequential		Counterfactual		Descriptive
Model	tIoU	$\mathcal{J}\&\mathcal{F}$	tIoU	$\mathcal{J}\&\mathcal{F}$	tIoU	$\mathcal{J}\&\mathcal{F}$	tIoU	$\mathcal{J}\&\mathcal{F}$	tIoU	$\mathcal{J}\&\mathcal{F}$
Different Training Strategies
SFT only	44.29	43.76	44.89	43.73	42.03	44.50	41.00	39.71	48.94	46.89
SFT + chain-aligned RL	50.20	53.67	50.57	52.95	48.08	57.04	46.99	50.82	54.94	53.86
Different Reward Functions
w/o $\mathcal{R}_{ta}$ and $\mathcal{R}_{stg}$	47.72	47.90	48.22	46.50	44.74	51.13	44.23	43.24	53.42	50.85

Appendix

Table 8: Temporal IoU and segmentation performance on GroundMoRe.

Model	MLLM	Ref-DAVIS17			Ref-YouTube-VOS
Model	MLLM	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
Veason-R1-3B † ( Gong et al., 2026 )	Qwen2.5-VL-3B	71.2	68.3	74.2	66.6	64.9	68.2
VideoSEG-O3-2B ( Dai et al., 2026 )	Qwen3-VL-2B	80.0	76.0	84.0	70.5	68.3	72.7
VideoSEG-O3-4B ( Dai et al., 2026 )	Qwen3-VL-4B	79.4	75.5	83.2	74.1	71.5	76.6
TCoVP-3B † (Ours)	Qwen2.5-VL-3B	79.3	75.6	83.1	72.6	71.1	74.1
TCoVP-4B † (Ours)	Qwen3-VL-4B	81.7	78.1	85.4	73.6	71.7	75.4

Appendix

Table 9: Performance on conventional referring video object segmentation benchmarks. † denotes zero-shot transfer, where the training split of the corresponding evaluation benchmark is not used.

Figure 7: Qualitative comparison with VideoSEG-O3-4B. VideoSEG-O3-4B identifies the relevant shot segment but produces a mask that follows the player rather than the queried basketball. TCoVP-4B explicitly distinguishes the contextual participants from the basketball, grounds the target over the relevant interval, and provides timestamped bounding-box prompts for mask propagation.

Figure 8: Zero-target recognition. TCoVP-3B identifies that the queried turtle is absent from the video, whereas Veason-R1-3B hallucinates the target and produces an erroneous mask trajectory.

Figure 9: Relational and action-based grounding. TCoVP-3B tracks the basketball during the relevant action interval and answers the query correctly, while Veason-R1-3B incorrectly grounds the acting person.

Figure 10: Failure case caused by relational-role confusion. TCoVP-3B correctly identifies and tracks both players but reverses their roles in the query, selecting the man in white instead of the ground-truth target, the man in red.

Temporal Reasoning in VLMs

See all 23 papers

RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation
May 8, 2026Junwei Wen, Deshui Miao, Guangming Lu +2Video Object Segmentation Video Reasoning
Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · Pazhou Lab (Huangpu)
VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation
Mar 28, 2026Jihwan Hong, Jaeyoung DoVideo Object Segmentation Referring Video Object Segmentation
AIDAS Laboratory, 1IPAI, Seoul National University · AIDAS Laboratory, 2ECE, Seoul National University
VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation
Jun 5, 2026Ming Dai, Sen Yang, Boqiang Duan +4Video Object Segmentation Referring Video Object Segmentation
School of Automation, Southeast University, Nanjing, China · Baidu Inc, Beijing, China · College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, China

May 8, 2026J/K move · Enter open · S save

RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation

Junwei Wen, Deshui Miao, Guangming Lu, Xin Li+1

Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · Pazhou Lab (Huangpu)

Video Object Segmentation Video Reasoning Video Frame Selection Reasoning Segmentation RL for VLMs

Open paper AskPDF arXiv

Video Reasoning Segmentation (VRS) aims to segment target objects in videos based on implicit instructions that convey human intent and temporal logic. Existing MLLM-based methods predict masks with a [SEG] token after selecting frames via simple sampling or an auxiliary MLLM, where limited supervision and frame-language similarity rules often yield narrow-scope keyframe choices that weaken holistic temporal understanding and lead to brittle localization in complex multi-object scenes. To address these issues, we introduce RCoT-Seg, a video-of-thought framework that factorizes VRS into temporal video reasoning (TVR) and keyframe target perception (KTP), explicitly separating temporal reasoning from spatial perception. Specifically, in the TVR stage, an agentic keyframe selection module, initialized with a curated CoT-start corpus and refined by GRPO under task-aligned rewards, is proposed to generate and reselect the keyframe through self-evaluation, strengthening moment localization and temporal reasoning. In the KTP stage, RCoT-Seg performs high-resolution segmentation on the selected frame and propagates masks with SAM2-based methods across the sequence, replacing heuristic sampling and external selectors while improving spatial precision and inter-frame consistency. Extensive experimental results demonstrate that the proposed RCoT-Seg achieves favorable performance against the state-of-the-art methods. The code and models will be publicly released at https://github.com/Victor-wjw/RCoT-Seg.

Abstract

Temporal Reasoning in VLMs

Explore similar work

RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation

VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation

VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation