Unlocking Motion in Expressions: Temporal Calibration for Referring Video Object Segmentation

Yiwen Jiang; Zhengtong Zhu; Ruixin Zhang; Jiaqing Fan

cs.CVAug 17, 2026

Unlocking Motion in Expressions: Temporal Calibration for Referring Video Object Segmentation

Authors: Yiwen Jiang, Zhengtong Zhu, Ruixin Zhang, Jiaqing Fan

Organizations: School of Computer Science & Technology Soochow University Suzhou, China

Abstract

Referring Video Object Segmentation (RVOS) aims to segment referred objects at the pixel level in video sequences based on natural language descriptions. Existing methods typically introduce motion information within a unified cross-modal temporal modeling framework, where language cues are used for target localization and segmentation. However, the dependency of expressions on motion semantics is not explicitly modeled, making it difficult to adaptively adjust the use of motion information according to different semantic requirements. To address these issues, we propose an Expression-driven Motion Calibration (EMC) framework for RVOS that explicitly unlocks and leverages the motion semantics within expressions. The proposed method extracts interpretable motion control signals from expressions via a Motion Signal Processing (MSP) module, and employs a Motion Influence Calibration (MIC) module to adjust the contribution of motion cues during temporal decision making. In addition, a Semantic Temporal Stage Construction (STSC) module is introduced to build expression-relevant temporal stages, providing a compact temporal candidate space for motion calibration. Through extensive evaluation on six standard benchmarks, including Ref-YouTubeVOS, Ref-DAVIS17, MeViS (valid/valid $^u$ ), A2D-Sentences, and JHMDB-Sentences, the superiority of our method is validated. We will release the code on https://github.com/Jeven7/EMC.

Figures & tables

Figure 1. Overview of EMC. (a) Existing RVOS methods typically apply motion modeling to entire video sequences, incorporating motion cues into expressions without explicit differentiation. (b) Our method models expression-related motion importance as motion signals and calibrates motion effects within the local structured semantic temporal space.

Figure 2. The architecture of EMC. First, STSC constructs expression-related temporal phases based on input video and expressions, defining a compact and efficient temporal influence space for subsequent adaptive temporal decision making process. Simultaneously, MSP models the importance of motion semantics within gesture expressions to extract explicit motion control signals. Finally, MIC calibrates the contribution of motion cues during the temporal decision making process. Through this approach, motion information can be utilized selectively and explicitly according to the semantic requirements of different expressions, thereby more effectively enhancing the robustness and flexibility of the RVOS system in complex scenarios.

Method	Publication	Ref-YouTube-VOS			Ref-DAVIS17			MeViS
Method	Publication	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
URVOS ( Seo et al., 2020 )	ECCV 2020	47.2	45.3	49.2	51.6	47.3	56.0	31.0	29.8	32.2
ReferFormer ( Wu et al., 2022 )	CVPR 2022	62.9	61.3	64.6	61.1	58.1	64.1	27.8	25.7	29.9
MTTR ( Botach et al., 2022 )	CVPR 2022	55.3	54.0	56.6	-	-	-	-	-	-
HTML ( Han et al., 2023 )	ICCV 2023	63.4	61.6	65.2	62.1	59.2	65.1	-	-	-
SgMg ( Miao et al., 2023 )	ICCV 2023	65.7	63.9	67.4	63.3	60.6	66.0	-	-	-

Table 1. Performance comparison with previous methods on Ref-YouTube-VOS, Ref-DAVIS17 and MeViS. The highest result in each column will be marked in bold , and the second-highest result will be underlined .

Method	Publication	MeViS (valid u )
Method	Publication	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
LMPM ( Ding et al., 2023 )	ICCV 2023	40.2	36.5	43.9
DsHmp ( He and Ding, 2024 )	CVPR 2024	55.3	51.0	60.4
VISA-7B ( Yan et al., 2024a )	ECCV 2024	51.2	48.0	54.4
DMVS ( Fang et al., 2025 )	CVPR 2025	58.3	52.6	63.9
SSA ( Pan et al., 2025 )	CVPR 2025	57.8	52.3	63.1

Table 2. Performance comparison with previous methods on the valid u set of MeViS.

Method	A2D-Sentences		JHMDB-Sentences
Method	oIoU	mIoU	oIoU	mIoU
ReferFormer ( Wu et al., 2022 )	78.6	70.3	73	71.8
SgMg ( Miao et al., 2023 )	79.9	72	73.7	72.5
SOC ( Luo et al., 2023 )	80.7	72.5	73.6	72.3
WaveCL ( Chen et al., 2025 )	79.2	71.0	73.0	71.8
DAVLS ( Cao et al., 2026 )	80.9	72.4	73.6	72.5

Table 3. Performance comparison with previous methods on A2D-Sentences and JHMDB-Sentences.

ID	Components			MeViS
ID	STSC	MSP	MIC	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
1	✗	✗	✗	52.6	49.8	55.3
2	✓	✗	✗	53.3	50.5	56.1
3	✗	✓	✓	53.7	50.9	56.5
4	✓	✓	✓	54.9	52.2	57.6

Table 4. Ablation study of main components of EMC.

$\alpha$	MeViS
$\alpha$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
0.05	54.7	52.0	57.5
0.1	54.9	52.2	57.6
0.2	54.7	52.0	57.4
0.3	54.8	52.1	57.5

Table 5. Ablation study of hyperparameter

\alpha

Figure 3. Ablation study on number of stages.

ID	Components			MeViS
ID	Visual	Text	Time	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
1	✓	✗	✗	54.5	51.8	57.2
2	✗	✓	✗	54.4	51.7	57.1
3	✓	✗	✓	54.5	51.7	57.2
4	✓	✓	✗	53.7	50.9	56.4
5	✗	✓	✓	54.5	51.9	57.2

Table 6. Ablation study of similarity components of STSC.

Figure 4. Performance under static and dynamic expressions.

Figure 5. Qualitative results on the Ref-YouTube-VOS (a) and MeVis (b) datasets. Time steps are directed from left to right.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	#Samples	Acc (%)	Prec (%)	Rec (%)	F1 (%)
Ref-DAVIS17	100	96.0	90.6	96.7	93.5
Ref-YouTube-VOS	150	96.7	92.3	100.0	96.0
MeViS	200	95.5	97.9	95.9	96.9

Appendix

Table 7. Reliability evaluation of MSP.

Dataset	Over- detection	Weak motion	Implicit / Ambiguous	Total
Ref-DAVIS17	3	0	1	4
Ref-YouTube-VOS	5	0	0	5
MeViS	3	3	3	9
Total	11	3	4	18

Appendix

Table 8. Error type distribution across datasets.

Figure 6. Distribution of motion strengths across datasets.

Type	Expression	MSP Prediction	Human Annotation	Analysis
Weak motion	the rabbit that is eating	Static	Dynamic	subtle motion is overlooked
Implicit motion	the last sheep to appear	Static	Dynamic	motion is implied but not explicit
Over-detection	a guitar being played by second man	Dynamic	Static	verb triggers false motion detection
Ambiguous action	monkey with a monkey on its back	Dynamic	Static	does not necessarily imply motion
Semantic ambiguity	a cow is standing far behind	Dynamic	Static	misinterpreted as motion-related context

Appendix

Table 9. Representative error cases of MSP.

A2D-Sentences
Method	Precision
Method	$[email protected]$	$[email protected]$	$[email protected]$	$[email protected]$	$[email protected]$
Gavrilyuk et al. ( Gavrilyuk et al., 2018 )	47.5	34.7	21.1	8.0	0.2
MTTR ( Botach et al., 2022 )	75.4	71.2	63.8	48.5	16.9
ReferFormer ( Wu et al., 2022 )	83.1	80.4	74.1	57.9	21.2
SOC ( Luo et al., 2023 )	82.1	82.7	76.5	60.7	25.2

Appendix

Table 10. Precision comparison with previous methods on A2D-Sentences and JHMDB-Sentences.

Figure 7. Visualization of Semantic Temporal Stages.

Figure 8. Ablation study on number of stages on MeViS.

$\alpha$	Ref-DAVIS17
$\alpha$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
0.1	77.56	74.41	80.71
0.2	77.57	74.42	80.72
0.3	77.63	74.49	80.77
0.4	77.65	74.51	80.79

Appendix

Table 11. Ablation study of motion calibration strength on Ref-DAVIS17.

	Ref-YouTube			Ref-DAVIS17			MeViS
$\beta$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
0	71.8	69.8	73.9	75.5	71.7	79.3	52.2	49.3	55.1
0.25	74.4	72.4	76.3	77.5	74.3	80.7	54.8	52.1	57.5
0.5	74.7	72.6	76.8	77.7	74.5	80.8	54.9	52.2	57.6
0.75	74.2	72.3	76.1	77.1	73.9	80.3	54.5	51.8	57.2
1	74.1	72.2	76.0	77.1	74	80.2	54.4	51.7	57.1

Appendix

Table 12. Sensitivity analysis of

\beta

Setting	FPS (frames/s)	Runtime (s/pair)	Memory (GB)
MSP	16.81	1.57	15.0
STSC	20.11	1.31	1.3
EMC	7.34	6.73	10.0

Appendix

Table 13. Inference efficiency of EMC.

Figure 9. Prompt and case for Motion Signal Processing.

Figure 10. Cases for Motion Signal Processing.

Figure 11. Visualization for multiple text references.

Figure 12. Visualization for multiple text references.

Referring Video Object Segmentation

Referring video object segmentation grounds language descriptions in video frames and produces temporally consistent masks for the described objects.

See all 43 papers

Weakly-Supervised Referring Video Object Segmentation through Text Supervision
Apr 20, 2026Miaojing Shi, Jun Huang, Zijie Yue +1Video Object Segmentation Referring Video Object Segmentation
College of Electronic and Information Engineering, Tongji University
VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation
Mar 28, 2026Jihwan Hong, Jaeyoung DoVideo Object Segmentation Referring Video Object Segmentation
AIDAS Laboratory, 1IPAI, Seoul National University · AIDAS Laboratory, 2ECE, Seoul National University
AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation
Mar 24, 2026Woojeong Jin, Jaeho Lee, Heeseong Shin +3Referring Video Object Segmentation Video Reasoning
KAIST AI

Apr 20, 2026J/K move · Enter open · S save

Weakly-Supervised Referring Video Object Segmentation through Text Supervision

Miaojing Shi, Jun Huang, Zijie Yue, Hanli Wang

College of Electronic and Information Engineering, Tongji University

Video Object Segmentation Referring Video Object Segmentation Weakly Supervised Learning

Open paper AskPDF arXiv

Referring video object segmentation (RVOS) aims to segment the target instance in a video, referred by a text expression. Conventional approaches are mostly supervised learning, requiring expensive pixel-level mask annotations. To tackle it, weakly-supervised RVOS has recently been proposed to replace mask annotations with bounding boxes or points, which are however still costly and labor-intensive. In this paper, we design a novel weakly-supervised RVOS method, namely WSRVOS, to train the model with only text expressions. Given an input video and the referring expression, we first design a contrastive referring expression augmentation scheme that leverages the captioning capabilities of a multimodal large language model to generate both positive and negative expressions. We extract visual and linguistic features from the input video and generated expressions, then perform bi-directional vision-language feature selection and interaction to enable fine-grained multimodal alignment. Next, we propose an instance-aware expression classification scheme to optimize the model in distinguishing positive from negative expressions. Also, we introduce a positive-prediction fusion strategy to generate high-quality pseudo-masks, which serve as additional supervision to the model. Last, we design a temporal segment ranking constraint such that the overlaps between mask predictions of temporally neighboring frames are required to conform to specific orders. Extensive experiments on four publicly available RVOS datasets, including A2D Sentences, J-HMDB Sentences, Ref-YouTube-VOS, and Ref-DAVIS17, demonstrate the superiority of our method. Code is available at https://github.com/viscom-tongji/WSRVOS.

Abstract

Referring Video Object Segmentation

Explore similar work

Weakly-Supervised Referring Video Object Segmentation through Text Supervision

VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation

AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation