VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation

Jihwan Hong; Jaeyoung Do

cs.CVMar 28, 2026

VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation

Authors: Jihwan Hong, Jaeyoung Do

Organizations: AIDAS Laboratory, 1IPAI, Seoul National University · AIDAS Laboratory, 2ECE, Seoul National University

Abstract

Referring Video Object Segmentation (RVOS) aims to segment target objects in videos based on natural language descriptions. However, fixed keyframe-based approaches that couple a vision language model with a separate propagation module often fail to capture rapidly changing spatiotemporal dynamics and to handle queries requiring multi-step reasoning, leading to sharp performance drops on motion-intensive and reasoning-oriented videos beyond static RVOS benchmarks. To address these limitations, we propose VIRST (Video-Instructed Reasoning Assistant for Spatio-Temporal Segmentation), an end-to-end framework that unifies global video reasoning and pixel-level mask prediction within a single model. VIRST bridges semantic and segmentation representations through the Spatio-Temporal Fusion (STF), which fuses segmentation-aware video features into the vision-language backbone, and employs the Temporal Dynamic Anchor Updater to maintain temporally adjacent anchor frames that provide stable temporal cues under large motion, occlusion, and reappearance. This unified design achieves state-of-the-art results across diverse RVOS benchmarks under realistic and challenging conditions, demonstrating strong generalization to both referring and reasoning oriented settings. The code and checkpoints are available at https://github.com/AIDASLab/VIRST.

Figures & tables

Figure 1: Performance comparison with existing RVOS methods. VIRST achieves state-of-the-art results across all referring video object segmentation benchmarks, while maintaining competitive performance on referring and reasoning-based image segmentation tasks.

Figure 2: Overall architecture of VIRST. (a) VIRST utilizes a VLM to capture global video context and identify query-aligned targets. The Spatio-Temporal Fusion (STF) fuses features from the segmentation-aware vision encoder, while the Temporal Dynamic Anchor Updater (TDAU) provides local and long-range temporal cues through a dual-track memory design. (b) The ST-Fusion module includes an Initial ST-Fusion stage, where the [ST] tokens are fused with segmentation-aware video tokens prior to VLM processing, followed by the Second ST-Fusion stage that applies cross-attention between the temporally expanded [ST] tokens and the segmentation-aware video tokens. The resulting spatiotemporal prompts are sliced to produce frame-specific segmentation prompts.

Figure 3: Anchor selection scheme of TDAU. Given a video, TDAU selects anchor-frame candidates $\mathcal{A}$ and generates their segmentation masks using frame-wise segmentation prompts. For each non-anchor frame, the module retrieves the $\alpha$ temporally nearest anchor frames $\mathcal{I}^{(k)}_{\text{Anchor}}$ . The anchor set is temporally updated over time as the video advances. This strategy maintains temporal locality while ensuring coverage of a broader temporal range.

Figure 4: Qualitative results of VIRST. Across diverse video segmentation scenarios, VIRST generates high-quality masks despite strong distractors, reasoning-oriented queries, heavy occlusions, small objects, and multiple interacting instances—demonstrating robust spatiotemporal reasoning and effective integration of both global and local video context. Results are best viewed when zoomed in.

Category	Model	Venue	Referring			Reasoning			Overall			$\mathcal{R}$
Category	Model	Venue	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{R}$
Segmentation Expert	MTTR [ 2 ]	ECCV’22	29.8	30.2	30.0	20.4	21.5	21.0	25.1	25.9	25.5	5.6
	ReferFormer [ 55 ]	CVPR’22	31.2	34.3	32.7	21.3	25.6	23.4	26.2	29.9	28.1	8.8
	LMPM [ 9 ]	ICCV’23	29.0	39.1	34.1	13.3	24.8	19.0	21.2	27.1	26.8	3.8
MLLM-based Segmentation Method	LISA-7B [ 24 ]	CVPR’24	44.3	47.1	45.7	33.8	38.4	36.1	39.1	42.7	40.9	9.3
	VISA-7B [ 57 ]	ECCV’24	49.2	52.6	50.9	40.6	45.4	43.0	44.9	49.0	46.9	15.5

Table 1: Performance comparison with previous methods on the ReVOS benchmark. Best results are in bold ; second-best are underlined .

\mathcal{R}

is the robustness score for hallucination evaluations [ 57 , 28 ] .

Category	Model	Venue	MeViS			Ref-YT-VOS			Ref-DAVIS17
Category	Model	Venue	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
Segmentation Expert	ReferFormer [ 55 ]	CVPR’22	29.8	32.2	31.0	61.3	64.6	62.9	58.1	64.1	61.1
	OnlineRefer [ 54 ]	ICCV’23	-	-	-	61.6	65.5	63.5	61.6	67.7	64.8
	SAMWISE [ 8 ]	CVPR’25	49.5	46.6	52.4	69.2	67.8	70.6	70.6	67.4	74.5
	MPG-SAM2 [ 43 ]	ICCV’25	50.7	56.7	53.7	71.7	76.1	73.9	68.8	76.0	72.4
	ReferDINO [ 31 ]	ICCV’25	44.7	53.9	49.3	67.0	71.5	69.3	65.1	72.9	68.9

Table 2: Performance comparison with previous methods on the validation sets of RVOS datasets. The best results are shown in bold , and the second-best results are underlined .

Model	RefCOCO			RefCOCO+			RefCOCOg		ReasonSeg
Model	val	testA	testB	val	testA	testB	val(U)	test(U)	gIoU	cIoU
LISA-7B [ 24 ]	74.9	79.1	72.3	65.1	70.8	58.1	67.9	70.6	52.9	54.0
PixelLM-7B [ 42 ]	73.0	76.5	68.2	66.3	71.7	58.3	69.3	70.5	-	-
GLaMM [ 40 ]	79.5	83.2	76.9	72.6	78.7	64.6	74.2	74.9	-	-
VISA-7B [ 57 ]	72.4	75.5	68.1	59.8	64.8	53.1	65.5	66.4	52.7	57.8
VRS-HQ-7B [ 16 ]	73.5	77.5	69.5	61.7	67.6	64.3	66.7	67.5	51.7	52.9

Table 3: Performance comparison with previous MLLM-based segmentation methods on referring and reasoning image segmentation datasets. The best results are shown in bold , and the second-best results are underlined .

Figure 5: STF patch-wise attention visualization. We visualize the $8\times 8$ patch-level attention from the STF before feeding it into the segmentation decoder. The attention maps consistently highlight key motion regions along the spatiotemporal dimension.

Initial ST-Fusion	Second ST-Fusion	MeViS
Initial ST-Fusion	Second ST-Fusion	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
✓	✓	60.4	65.4	62.9
✓		57.1	62.4	59.7
		57.0	61.8	59.4

Table 4: Ablation study of the Initial and Second ST-Fusion modules on the MeViS validation set. ✓ indicates that the corresponding module is enabled.

Anchor Frame Selection Strategy	MeViS
Anchor Frame Selection Strategy	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
Dynamic anchor selection	60.4	65.4	62.9
First-frame baseline	55.4	60.4	57.9
CLIP-guided selection	56.8	61.8	59.3
Random-3 sampling	59.6	64.8	62.2
Uniform-3 sampling	59.4	64.3	61.9

Table 5: Ablation study of anchor frame selection strategies on the MeViS validation set.

Number of Anchor Frames ( $\alpha$ )	MeViS
Number of Anchor Frames ( $\alpha$ )	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
2	60.1	65.0	62.5
3	60.4	65.4	62.9
4	60.4	65.4	62.9
6	60.6	65.5	63.0
8	60.7	65.8	63.2

Table 6: Ablation study on the number of anchor frames (

\alpha

) used in VIRST on the MeViS validation set.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Dataset
RVOS	Ref-DAVIS17 [ 22 ]
	Ref-YouTube-VOS [ 45 ]
	MeViS [ 9 ]
	ReVOS [ 57 ]
Video Instance Segmentation	LV-VIS [ 48 ]
Referring Image Segmentation	RefCOCO [ 21 ]

Appendix

Table 7: Datasets used for training VIRST.

Stage	MeViS (valid_u)
Stage	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
Stage 1	57.6	63.6	60.6
Stage 2	61.1	67.8	64.4
Stage 3	69.6	75.7	72.6

Appendix

Table 8: Performance after each training stage on MeViS (valid_u).

Training Stages	MeViS (valid_u)
Training Stages	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$
Full Pipeline (Stage 1+2+3)	69.6	75.7	72.6
Without Stage 1 (Stage 2+3)	62.8	68.8	65.8
Stage 3 Only	62.4	69.1	65.8

Appendix

Table 9: Ablation on training stage combinations on MeViS (valid_u).

Figure 6: Frame-Aware Video Tokenizer architecture.

$\alpha$	Efficiency
$\alpha$	FPS $\uparrow$	Memory (GB) $\downarrow$
2	5.14	37.52
4	5.04	37.52
6	4.98	37.52

Appendix

Table 10: Efficiency of VIRST with different

\alpha

Method	FPS $\uparrow$
VISA-7B [ 14 ] w/o postproc.	1.47
VRS-HQ-7B [ 12 ]	3.81
HyperSeg-3B [ 13 ]	1.54
VIRST-7B ( $\alpha=3$ , Ours)	5.10

Appendix

Table 11: Inference speed comparison across methods.

Figure 7: Qualitative results of VIRST on videos. Results are best viewed when zoomed in.

Figure 8: Qualitative results of VIRST on images. Results are best viewed when zoomed in.

Figure 9: Qualitative video understanding results of VIRST.

Video Object Segmentation

Video object segmentation assigns pixel-level masks to objects across frames while preserving their identities through motion, occlusion, and change.

See all 110 papers

Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation
Oct 18, 2026Junwei Wen, Deshui Miao, Guangming Lu +3Temporal Reasoning in VLMs Video Object Segmentation
Harbin Institute of Technology, Shenzhen · University of California at Merced · Pengcheng Laboratory +1
RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation
May 8, 2026Junwei Wen, Deshui Miao, Guangming Lu +2Video Object Segmentation Video Reasoning
Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · Pazhou Lab (Huangpu)
VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation
Jun 5, 2026Ming Dai, Sen Yang, Boqiang Duan +4Video Object Segmentation Referring Video Object Segmentation
School of Automation, Southeast University, Nanjing, China · Baidu Inc, Beijing, China · College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, China

Oct 18, 2026J/K move · Enter open · S save

Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation

Junwei Wen, Deshui Miao, Guangming Lu, Ming-Hsuan Yang+2

Harbin Institute of Technology, Shenzhen · University of California at Merced · Pengcheng Laboratory+1

Temporal Reasoning in VLMs Video Object Segmentation CoT Reasoning

Open paper AskPDF arXiv

Video Reasoning Segmentation (VRS) requires a model to identify an implicitly described target and predict its pixel-level mask trajectory, while resolving the spatial relations and temporal events that determine the answer. Text-only Chain-of-Thought (CoT) can explain a target choice without making the underlying visual references explicit, leaving a video reference gap between linguistic inference and the evidence distributed across frames. We introduce Temporal Chain of Video Primitives (TCoVP), a spatiotemporally grounded form of Video CoT that organizes query-relevant observations into a chronological reasoning chain. Each primitive associates an entity reference with temporal support and timestamped spatial evidence, and the chain interleaves these references with language to guide reasoning about which entity is relevant, when the queried event holds, and where its evidence appears. The selected target primitives then provide box prompts and temporal support for video mask prediction. We construct TCoVP-SFT-15k and TCoVP-RL-13k to learn this representation through supervised fine-tuning and chain-aligned reinforcement learning. Extensive experiments across five VRS benchmarks and three model backbones demonstrate the strong performance of TCoVP, highlighting the effectiveness of organizing spatial and temporal evidence within the reasoning process. All models and code will be publicly released. Project page: https://victor-wjw.github.io/TCoVP/

Abstract

Video Object Segmentation

Explore similar work

Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation

RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation

VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation