Show Me When and Where: Towards Referring Video Object Segmentation in the Wild

Mingqi Gao; Jinyu Yang; Jingnan Luo; Xiantong Zhen; Jungong Han; Giovanni Montana; Feng Zheng

cs.CVMar 15, 2026

Show Me When and Where: Towards Referring Video Object Segmentation in the Wild

Authors: Mingqi Gao, Jinyu Yang, Jingnan Luo, Xiantong Zhen, Jungong Han, Giovanni Montana, Feng Zheng

Organizations: Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China · Warwick Manufacturing Group, University of Warwick, Coventry CV1 7AL, U.K. · Tapall.ai, Shenzhen 518055, China · Central Research Institute, United Imaging Healthcare, Co., Ltd., Shanghai, 201807, China ·

Abstract

Referring video object segmentation (RVOS) has recently generated great popularity in computer vision due to its widespread applications. Existing RVOS setting contains elaborately trimmed videos, with text-referred objects always appearing in all frames, which however fail to fully reflect the realistic challenges of this task. This simplified setting requires RVOS methods to only predict where objects, with no need to show when the objects appear. In this work, we introduce a new setting towards in-the-wild RVOS. To this end, we collect a new benchmark dataset using Youtube Untrimmed videos for RVOS - YoURVOS, which contains 1,120 in-the-wild videos with 7 times more duration and scenes than existing datasets. Our new benchmark challenges RVOS methods to show not only where but also when objects appear in videos. To set a baseline, we propose Object-level Multimodal TransFormers (OMFormer) to tackle the challenges, which are characterized by encoding object-level multimodal interactions for efficient and global spatial-temporal localisation. We demonstrate that previous VOS methods struggle on our YoURVOS benchmark, especially with the increase of target-absent frames, while our OMFormer consistently performs well. Our YoURVOS dataset offers an imperative benchmark, which will push forward the advancement of RVOS methods for practical applications.

Figures & tables

Fig. 1: Examples from YoURVOS. The text-referred objects are highlighted spatially in blue and orange masks and temporally in thick lines in the same colours. The most unique feature of YoURVOS is untrimmed videos, i.e., all videos are not trimmed to fit the span of any text-referred object. As shown in examples, most objects in YoURVOS appear in one or several segments in a video rather than all frames. With the untrimmed setting, more complex situations are also considered, e.g., broader temporal contexts and multiple scenes. These promote RVOS closer to realistic scenarios and bring several new challenges: (1) spatial-temporal joint localisation, (2) multimodal analysis on untrimmed videos, and (3) long-term video segmentation.

Dataset	Year	Video	Object	Exp	Scene	Dur (e)	Dur (v)	TI
A2D-Sentences [ 4 ]	2018	3,782	4,825	6,656	1.00	–	4.91s	–
J-HMDB-Sentences [ 4 ]	2018	928	928	928	1.00	1.37s	1.37s	0.00%
Ref-DAVIS [ 5 ]	2018	140	255	1,644	1.00	3.11s	3.45s	3.21%
Ref-YouTube-VOS [ 1 ]	2020	3,978	7,451	15,009	1.00	4.95s	5.45s	6.97%
MeViS [ 6 ]	2023	2,006	8,171	28,570	1.63	10.88s	13.16s	15.80%
ReVOS [ 7 ]	2024	1,042	47,134	35,074	1.75	10.15s	13.37s	16.23%

TABLE I: Comparisons between YoURVOS and previous RVOS datasets. Scene (v) and Dur (v) are the number of scenes (measured by PySceneDetect [ 17 ] ) and the duration per video (measured in seconds, abbreviated as s in the table). Dur (e) and TI are the duration and target-irrelevant rate per expression. Due to sparse annotations, Dur (e) and TI cannot be measured on A2D-Sentences. The largest metrics are presented in bold .

Fig. 2: Dataset construction pipeline. (a) For Ref-YouTube-VOS videos (orange film), we retrieve their untrimmed sources from YouTube-8M [ 39 ] to pad before and after (red films); (b) We select target objects from collected videos and annotate corresponding language descriptions and spans (orange thick line); (c) We annotate masks (orange masks) for the target objects. From the example above, it is also clear that YoURVOS videos consider much more complex contexts and scenes than trimmed ones.

Fig. 3: Previous RVOS v.s. RVOS in the wild. Video frames are sampled every 3 seconds and with timestamps left-bottom. Yellow and white numbers indicate target-relevant and -irrelevant frames. Compared to previous benchmarks, YoURVOS is much more challenging and closer to realistic scenarios with long-term videos, target-irrelevant distractions, and multiple scenes, calling for spatial-temporal joint localisation.

Fig. 4: Distributions of RVOS datasets on object span, number of scenes, and TI. For each violin chart summarising start/end times of object appearance, shades depict the probability density. Three markers are minimum/median/maximum data. The reason for plots with less than three markers lies in the over-concentrative data distribution. Violin charts share the same legend as the line charts in this figure. A2D-Sentences [ 4 ] is not shown in some cases due to its sparse mask annotations.

Fig. 5: Framework of Object-level Multimodal transFormers (OMFormer).

Fig. 6: Spatial-Temporal Association in (a) Previous RVOS methods and (b) OMFormer. (c) Diagram of the Sequence and Relevance-Aware Decoder; (d) Quantitative comparisons between OMFormer and representative RVOS works in training costs.

Method	Year	Venue	Backbone (L)	Backbone (V)	$\mathcal{J}\&\mathcal{F}\uparrow$	$\mathcal{J}\uparrow$	$\mathcal{F}\uparrow$	tIoU $\uparrow$
ReferFormer [ 43 ]	2022	CVPR	RoBERTa-base	ResNet-50	12.0	12.1	11.9	32.2
				ResNet-101	22.3	22.2	22.3	33.4
				Swin-T	22.6	22.7	22.6	34.1
				Swin-L	24.9	24.6	25.2	34.4
				Video-Swin-T	23.0	22.8	23.1	33.7
				Video-Swin-B	24.6	24.3	24.8	34.5

TABLE II: Quantitative comparisons on YoURVOS. Backbone (L) denotes the Language Backbone, and Backbone (V) denotes the Vision Backbone. For the Vision Backbone, T, B, L, and H correspond to the Tiny, Big, Large, and Huge model variants. For the Language Backbone, B indicates the number of parameters in billions. The best and the second-best scores in each column are highlighted in bold and underlined , respectively.

Fig. 7: Impact of target-irrelevant frames on RVOS results (Left:

\mathcal{J}\&\mathcal{F}

; Right: tIoU). Horizontal axises represent the videos grouped by the TI rate.

Fig. 8: Qualitative results on YoURVOS. Blue and orange boxes and lines show text-referred objects spatially and temporally. Red masks are overlap masks.

D	O	$T_{\mathrm{train}}$	Mem	S	Overall (489)		0%-33% (88)		33%-66% (77)		66%-100% (324)
✓	✗	5	15.3	✗	27.5	32.9	79.8	90.4	41.2	50.0	9.9	13.3
		12	29.2	✗	23.8	32.6	80.0	90.9	28.9	48.9	7.3	12.9
		18	>32	–	–	–	–	–	–	–	–	–
✓	✓	5	16.5	✗	27.3	33.2	81.1	91.0	41.4	51.1	9.2	13.2
				✓	11.5	16.2	36.5	49.9	11.3	17.1	4.7	6.8
		12	>32	–	–	–	–	–	–	–	–	–

TABLE III: Ablations on spatial-temporal associations and span predictions. ‘D’: Dense; ‘O’: Object, ‘S’: Span predictions,

T_{\mathrm{train}}

: Number of training frames, ‘Mem’: Memory cost (GB) during training. Scores: Average

\mathcal{J}\&\mathcal{F}

(left) and tIoU (right) over TI-grouped videos. Numbers in brackets: Number of expressions in each group.

$T_{\mathrm{train}}$	Memory	Overall (489)		0%-33% (88)		34%-66% (67)		37%-100% (324)
20	24.0	30.6	41.0	63.3	86.4	41.5	57.5	19.1	24.8
22	26.7	30.3	40.7	62.9	86.3	42.8	59.3	18.4	23.9
24	29.8	33.7	44.9	64.5	87.1	44.7	62.1	22.7	29.3
26	30.9	32.5	43.2	64.6	86.6	43.4	60.2	21.1	27.3
28	32.0	34.7	46.6	65.6	88.8	44.1	61.2	23.9	31.7
30	34.2	34.5	45.9	66.5	89.3	45.9	63.5	23.0	29.9

TABLE IV: More ablations on

T_{\mathrm{train}}

. ‘Memory’: Memory cost (GB) during training. Scores: Average

\mathcal{J}\&\mathcal{F}

(left) and tIoU (right) over TI-grouped videos. Numbers in brackets: Number of expressions in each group.

	Overall (489)		0%-33% (88)		33%-66% (77)		66%-100% (324)
w/o Rel	32.9	44.0	64.9	87.9	43.4	60.9	21.6	28.1
w/ Rel	33.7	44.9	64.5	87.1	44.7	62.1	22.7	29.3

TABLE V: Ablations on relevance (Rel) predictions.

	Overall (489)		0%-33% (88)		33%-66% (77)		66%-100% (324)
w/Decoder	27.9	37.8	49.2	69.9	35.3	48.9	20.3	26.5
Coupled	21.7	29.9	49.6	69.0	33.7	48.6	11.3	14.9
Decoupled	33.7	44.9	64.5	87.1	44.7	62.1	22.7	29.3

TABLE VI: Ablations on Sequence and Relevance Decoder.

Method	Backbone	Train		Fine-tune
		$\mathcal{J}\&\mathcal{F}$	IoU	$\mathcal{J}\&\mathcal{F}$	IoU
ReferFormer [ 43 ]	Resnet-50	27.5	32.9	29.7 (+2.2)	34.1 (+1.2)
UNINEXT [ 51 ]	Resnet-50	26.9	32.7	29.1 (+2.2)	33.3 (+0.6)
R2VOS [ 52 ]	Resnet-50	27.2	33.1	29.3 (+2.1)	33.9 (+0.8)
MUTR [ 57 ]	Resnet-50	27.3	33.2	29.2 (+1.9)	34.0 (+0.8)
SAMWISE [ 64 ]	SAM-V2-B+	24.5	33.7	26.8 (+2.3)	34.1 (+0.4)

TABLE VII: Quantitative comparisons of OMFormer and existing SoTAs under different training schemes (‘Train’: same training setting as OMFormer, ‘Fine-tune’: pre-train on other RVOS datasets and fine-tune on YoURVOS), where OMFormer uses Ref-YouTube-VOS, others use more diverse ones.

Method	Backbone	R-YTV	MeViS	#Params	Memory	FPS
		$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	(MB)	(GB)
ReferFormer [ 43 ]	Resnet-50	58.7	31.0	131.1	20.5	3.3
UNINEXT [ 51 ]	Resnet-50	61.2	–	128.5	14.5	2.9
R2VOS [ 52 ]	Resnet-50	60.2	–	132.7	14.5	2.8
MUTR [ 57 ]	Resnet-50	61.9	–	138.5	28.1	3.3
DsHmp [ 58 ]	Swin-T	63.6	46.4	272.3	39.7	2.8

TABLE VIII: Comparisons of OMFormer and existing methods with the similar backbone on complexity (during inference) and the quantitative performance on other benchmarks.

Fig. 9: Failure cases of OMFormer. Orange boxes and lines highlight text-referred objects spatially and temporally. Orange masks are predictions.

Multimodal Transformer

See all 22 papers

AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation
Mar 24, 2026Woojeong Jin, Jaeho Lee, Heeseong Shin +3Referring Video Object Segmentation Video Reasoning
KAIST AI
ReflexTrack: A Feedback-Driven Agent for Training-Free Referring Video Object Segmentation
Jul 27, 2026Yuanjia Li, Tianyang Xu, Tao Zhou +3Video Object Segmentation Referring Video Object Segmentation
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, GU2 7XH, UK
Weakly-Supervised Referring Video Object Segmentation through Text Supervision
Apr 20, 2026Miaojing Shi, Jun Huang, Zijie Yue +1Video Object Segmentation Referring Video Object Segmentation
College of Electronic and Information Engineering, Tongji University

Mar 24, 2026J/K move · Enter open · S save

AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation

Woojeong Jin, Jaeho Lee, Heeseong Shin, Seungho Jang+2

KAIST AI

Referring Video Object Segmentation Video Reasoning

Open paper AskPDF arXiv

Referring Video Object Segmentation (RVOS) aims to segment a target object throughout a video given a natural language query. Training-free methods for this task follow a common pipeline: a MLLM selects keyframes, grounds the referred object within those frames, and a video segmentation model propagates the results. While intuitive, this design asks the MLLM to make temporal decisions before any object-level evidence is available, limiting both reasoning quality and spatio-temporal coverage. To overcome this, we propose AgentRVOS, a training-free agentic pipeline built on the complementary strengths of SAM3 and a MLLM. Given a concept derived from the query, SAM3 provides reliable perception over the full spatio-temporal extent through generated mask tracks. The MLLM then identifies the target through query-grounded reasoning over this object-level evidence, iteratively pruning guided by SAM3's temporal existence information. Extensive experiments show that AgentRVOS achieves state-of-the-art performance among training-free methods across multiple benchmarks, with consistent results across diverse MLLM backbones. Our project page is available at: https://cvlab-kaist.github.io/AgentRVOS/.

Abstract

Multimodal Transformer

Explore similar work

AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation

ReflexTrack: A Feedback-Driven Agent for Training-Free Referring Video Object Segmentation

Weakly-Supervised Referring Video Object Segmentation through Text Supervision