cs.CVApr 13, 2026

Online Reasoning Video Object Segmentation

Authors: Jinyuan Liu, Yang Wang, Zeyu Zhao, Weixin Li, Song Wang, Ruize Han

Organizations: Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced · School of Computer Science and Engineering, Beihang University · School of Electrical and Information Engineering, Northeast Petroleum University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reasoning video object segmentation predicts pixel-level masks in videos from natural-language queries that may involve implicit and temporally grounded references. However, existing methods are developed and evaluated in an offline regime, where the entire video is available at inference time and future frames can be exploited for retrospective disambiguation, deviating from real-world deployments that require strictly causal, frame-by-frame decisions. We study Online Reasoning Video Object Segmentation (ORVOS), where models must incrementally interpret queries using only past and current frames without revisiting previous predictions, while handling referent shifts as events unfold. To support evaluation, we introduce ORVOSB, a benchmark with frame-level causal annotations and referent-shift labels, comprising 210 videos, 12,907 annotated frames, and 512 queries across five reasoning categories. We further propose a baseline with continually-updated segmentation prompts and a structured temporal token reservoir for long-horizon reasoning under bounded computation. Experiments show that existing methods struggle under strict causality and referent shifts, while our baseline establishes a strong foundation for future research.

Figures & tables

Figure 1: Overview of ORVOSB data construction and annotation pipeline.
Figure 2: Illustrative examples of the five reasoning query types in ORVOSB.
Figure 3: Overview of the proposed online reasoning video object segmentation framework. At each time step, the model processes context frames and the current frame together with a dynamic token reservoir. The multimodal large language model produces segmentation tokens, which are adaptively fused to form a history-aware prompt for mask prediction. The fused token features are written back to the token reservoir, enabling structured long-term reasoning under strict causal constraints.
MethodsTypeAttr.SpatialActionInteract.Ext. Know.Overall
J&F\mathcal{J}\&\mathcal{F}J&F\mathcal{J}\&\mathcal{F}J&F\mathcal{J}\&\mathcal{F}J&F\mathcal{J}\&\mathcal{F}J&F\mathcal{J}\&\mathcal{F}J\mathcal{J}F\mathcal{F}J&F\mathcal{J}\&\mathcal{F}
VISA [ 49 ]Vid41.6%31.0%40.3%27.5%39.1%30.8%35.7%33.3%
VRS-HQ [ 15 ]Vid44.4%44.3%41.8%35.8%43.7%40.4%46.4%43.4%
GLUS [ 25 ]Vid43.0%45.8%47.0%32.9%51.1%42.7%47.7%45.2%
LISA [ 23 ]Img53.6%46.4%47.7%35.1%49.5%44.7%48.4%46.6%
VideoLISA [ 31 ]Vid53.2%52.1%46.1%34.0%55.0%47.5%53.1%50.3%
Table 1: Performance comparison with previous methods on the ORVOSB. Type indicates whether a method is image-based (Img) or video-based (Vid).
MethodsReVOSORVOSB
J\mathcal{J}F\mathcal{F}J&F\mathcal{J}\&\mathcal{F}J\mathcal{J}F\mathcal{F}J&F\mathcal{J}\&\mathcal{F}
Baseline49.5%54.5%52.0%49.2%55.6%52.4%
+ CUSP (w/o AGAF)50.5%55.4%53.0%51.8%58.3%55.1%
+ CUSP51.3%56.1%53.7%55.0%61.3%58.1%
+ CUSP + TR (US)51.8%56.4%54.1%58.0%63.9%60.9%
+ CUSP + TR (D2S)52.0%56.6%54.3%58.3%64.2%61.3%
Table 2: Ablation study on ReVOS and ORVOSB. CUSP refers to the continually-updating segmentation prompt, AGAF to affinity-guided adaptive fusion, TR to token reservoir, US to uniform sampling retention, and D2S to dense-to-sparse retention.
MethodsReferringReasoningOverall
J\mathcal{J}F\mathcal{F}J&F\mathcal{J}\&\mathcal{F}J\mathcal{J}F\mathcal{F}J&F\mathcal{J}\&\mathcal{F}J\mathcal{J}F\mathcal{F}J&F\mathcal{J}\&\mathcal{F}
LISA [ 23 ]44.3%47.1%45.7%33.8%38.4%36.1%39.1%42.7%40.9%
TrackGPT [ 57 ]46.7%49.7%48.2%36.8%41.2%39.0%41.8%45.5%43.6%
VISA [ 49 ]49.2%52.6%50.9%40.6%45.4%43.0%44.9%49.0%46.9%
GLUS [ 25 ]56.3%60.7%58.3%48.8%53.9%51.4%52.6%57.3%54.8%
VRS-HQ [ 15 ]59.8%64.5%62.1%53.5%58.7%56.1%56.6%61.6%59.1%
Table 3: Performance comparison with previous methods on the ReVOS dataset.
Figure 4: Qualitative comparison of segmentation results on ORVOSB and ReVOS.
More in this topic

Video Object Segmentation

Video object segmentation assigns pixel-level masks to objects across frames while preserving their identities through motion, occlusion, and change.

See all 110 papers

Explore similar work

CardsList

  1. VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation

    Jun 5, 2026Ming Dai, Sen Yang, Boqiang Duan +4Video Object SegmentationReferring Video Object Segmentation

    School of Automation, Southeast University, Nanjing, China · Baidu Inc, Beijing, China · College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, China

  2. VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation

    Mar 28, 2026Jihwan Hong, Jaeyoung DoVideo Object SegmentationReferring Video Object Segmentation

    AIDAS Laboratory, 1IPAI, Seoul National University · AIDAS Laboratory, 2ECE, Seoul National University

  3. FeVOS: Foresight Expression Video Object Segmentation

    Jun 24, 2026Kehan Lan, Kaining Ying, Henghui DingVideo Object SegmentationReferring Video Object Segmentation

    Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, China