Learning to Reason with Persistent Object States for Video Instance Segmentation

Yongxue Xu; Boxue Yang; Ziqian Liu; Shaoqiu Zhang; Rui Qian; Haopeng Chen

cs.CVSep 28, 2026

Learning to Reason with Persistent Object States for Video Instance Segmentation

Authors: Yongxue Xu, Boxue Yang, Ziqian Liu, Shaoqiu Zhang, Rui Qian, Haopeng Chen

Organizations: Shanghai Jiao Tong University · Sun Yat-sen University · Fudan University

Abstract

Video segmentation models maintain object identities by carrying instance information across frames. Under prolonged occlusion, reappearance, or interactions between similar instances, however, an unreliable update can overwrite a valid history and cause persistent identity drift. We introduce POSReasoner, a trainable, plug-and-play framework that explicitly decides when an observation should change an object's state. Each persistent state records identity, confidence, and absence history. A sparse state-observation graph supports Propose-Verify reasoning: provisional associations are revisited using object history, predicted presence, and competition among identities. The verified decisions determine whether to retain, update, reactivate, or suppress each state, while a learned gate controls the evidence written back to memory. Only verified transitions update the persistent state used in subsequent frames. POSReasoner uses standard video annotations and keeps the base model frozen, enabling integration with diverse VOS and VIS architectures. Experiments across long-term VOS and VIS benchmarks show consistent improvements over strong baselines, with the largest gains under occlusion and object reappearance.

Figures & tables

Figure 1: POSReasoner preserves identity under crowding and similar-instance interactions (a), and reactivates returning objects after long gaps or dynamic motion (b). Yellow denotes references; teal denotes predictions.

Figure 2: From association to persistent-state reasoning. (a) Frame-local association leaves ambiguous matches unresolved across time. (b) Direct write-back can propagate one unreliable observation through later states. (c) POSReasoner keeps identity-indexed states separate from host memory, verifies competing hypotheses, and applies an explicit state transition.

Figure 3: Overview of POSReasoner. (a) Frozen-host observations are paired with persistent object states. (b) Propose–Verify reasoning refines associations and actions using identity history and object competition. The shaded matrices associate identities with observations or null. (c) Verified decisions gate a single state update. Internal values are schematic; DAVIS frames with ground-truth masks illustrate the interface, not model predictions.

Backbone	Method	YTVIS 2019			YTVIS 2021			YTVIS 2022
Backbone	Method	AP	AP50	AP75	AP	AP50	AP75	AP	AP50	AP75
R50	GenVIS ( Heo et al., 2023 )	50.0	71.5	54.6	47.1	67.5	51.5	37.5	61.6	41.5
	CTVIS ( Ying et al., 2023 )	51.8	74.1	55.2	49.5	72.5	53.6	45.4	67.5	48.9
	+ POSReasoner	52.8	75.2	56.5	50.6	73.7	55.1	47.5	69.7	51.4
	DVIS++ ( Zhang et al., 2023b )	55.7	80.8	59.8	50.4	71.4	55.2	46.3	66.6	50.5
	+ POSReasoner	56.0	80.6	60.9	51.1	72.6	55.8	47.0	68.2	51.0

Table 1: YouTube-VIS validation. Published methods provide context; each highlighted row adds POSReasoner to the host immediately above. Bold and underline mark the best and second-best result within each backbone.

Figure 4: Qualitative comparison under occlusion, identity ambiguity, and object reappearance.

ResNet-50
Method	$\Delta$ AP	AP	AP50	AP75
CTVIS ( Ying et al., 2023 )	–	$34.6$	$59.9$	$33.8$
+ POSReasoner	+2.7	$37.3$	$63.2$	$37.0$
DVIS++ ( Zhang et al., 2023b )	–	$36.8$	$61.3$	$37.5$
+ POSReasoner	+2.1	38.9	$64.3$	39.4
DAQ ( Zhou et al., 2024 )	–	$38.3$	65.2	$37.3$

Table 2: OVIS validation. Highlighted rows add POSReasoner to the matched host;

\Delta

AP is relative to that host. Bold and underline mark the best and second-best results within each backbone.

Variant	State	React.	Cross	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
LVOS v1 cumulative components
SAM3 host ( Carion et al., 2025 )	✗	✗	✗	79.5	90.6	85.1
+ state reasoning	$\checkmark$	✗	✗	82.1	92.4	87.2
+ reactivation	$\checkmark$	$\checkmark$	✗	83.3	93.6	88.4
Full POSReasoner	$\checkmark$	$\checkmark$	$\checkmark$	83.4	93.7	88.6
LVOS v2 transfer

Table 3: Long-term VOS and cumulative component analysis with frozen SAM3 predictions.

Figure 5: LVOS v1: persistent state preserves identity through absence (top); reactivation reconnects it on return (bottom).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	AP	$\Delta$ AP	AP50	AP75
Frozen host	34.63	–	59.94	33.80
Host cues only	34.82	+0.18	60.21	34.09
Shuffled state	34.67	+0.04	59.95	33.94
State, no graph	36.55	+1.92	62.88	35.61
State, one step	36.13	+1.50	62.01	35.27
State, shared two-step	36.97	+2.33	63.00	36.17

Appendix

Table 4: Causal controls on OVIS with the same frozen candidates and training protocol.

Candidates	Params	Latency	Memory
10	0.90M	5.80 ms	12.71 MB
20	0.90M	5.66 ms	12.99 MB
40	0.90M	5.64 ms	14.03 MB

Appendix

Table 5: Five-member VIS reasoner efficiency on one A100, batch size 1.

Dataset	Gap	Events	Videos	Host $\mathcal{J}$	POSR $\mathcal{J}$	$\Delta\mathcal{J}$ [95% CI]	H/T/D
LVOS v1	5–10	45	16	70.6	74.9	$+4.3$ [ $-2.3$ , $16.3$ ]	22/9/14
	11–30	34	15	51.1	64.2	$+13.1$ [ $-3.1$ , $29.1$ ]	20/6/8
	31–100	14	8	72.2	78.0	$+5.9$ [ $-2.7$ , $19.1$ ]	7/3/4
	$>100$	6	4	23.6	30.6	$+7.1$ [ $-10.1$ , $46.9$ ]	1/2/3
LVOS v2	5–10	114	42	66.9	65.8	$-1.1$ [ $-7.7$ , $5.3$ ]	53/17/44
	11–30	83	39	56.6	61.1	$+4.5$ [ $-2.3$ , $11.6$ ]	38/15/30

Appendix

Table 6: Post-reappearance results by absence duration, with paired differences and 95% bootstrap confidence intervals.

Method	Backbone	AP	AP50	AP75
GenVIS ( Heo et al., 2023 )	ResNet-50	35.8	60.8	36.2
CTVIS ( Ying et al., 2023 )	ResNet-50	35.5	60.8	34.9
DVIS++ ( Zhang et al., 2023b )	ResNet-50	37.2	62.8	37.3
DAQ ( Zhou et al., 2024 )	ResNet-50	38.7	65.5	37.6
CTVIS	ResNet-50	34.6	59.9	33.8
CTVIS + POSReasoner	ResNet-50	37.3	63.2	37.0

Appendix

Table 7: Complete OVIS evaluation across three backbones. Cited rows are published results; each uncited host is paired with POSReasoner under the same frozen host and candidate set.

Query-based Association
Method	Backbone	YTVIS 2019			YTVIS 2021			YTVIS 2022
Method	Backbone	AP	AP50	AP75	AP	AP50	AP75	AP	AP50	AP75
MinVIS ( Huang et al., 2022 )	R50	47.4	69.0	52.1	44.2	66.0	48.1	23.3	47.9	19.3
VITA ∗ ( Heo et al., 2022 )	R50	49.8	72.6	54.5	45.7	67.4	49.5	32.6	53.9	39.3
GenVIS ( Heo et al., 2023 )	R50	50.0	71.5	54.6	47.1	67.5	51.5	37.5	61.6	41.5
Memory and Decoupled Tracking

Appendix

Table 8: Complete YouTube-VIS plug-in evaluation; 2022 baselines follow LOMM ( Lee et al., 2025a ) . Each POSReasoner row retains its frozen host; ∗ denotes offline refinement and

\dagger

provisional results.

Explore similar work

VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation
Jun 5, 2026Ming Dai, Sen Yang, Boqiang Duan +4Video Object Segmentation Chain-of-Thought Reasoning
School of Automation, Southeast University, Nanjing, China · Baidu Inc, Beijing, China · College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, China
MoVISA: Multi-Token Reasoning for Video Object Segmentation
Sep 24, 2026Ruining Zhao, Ho Kei Cheng, Alexander G SchwingVideo Object Segmentation Multimodal Large Language Models
University of Illinois Urbana-Champaign
QueenVIS: Rethinking Image-Only Training for Video Instance Segmentation via Query Enrichment
Jul 27, 2026Arian Kheirandish, Fardin Ayar, Ehsan Javanmardi +2Video Object Segmentation Long Videos
Amirkabir University of Technology (AUT), Tehran, Iran · The University of Tokyo, Tokyo, Japan

Jun 5, 2026, cs.CVJ/K move · Enter open · S save

VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation

Ming Dai, Sen Yang, Boqiang Duan, Boyuan Tong+3

School of Automation, Southeast University, Nanjing, China · Baidu Inc, Beijing, China · College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, China

Video Object Segmentation Chain-of-Thought Reasoning

Open paper AskPDF arXiv

Reasoning Video Object Segmentation (RVOS) demands a sophisticated integration of temporal dynamics, spatial details, and linguistic reasoning to achieve precise pixel-level localization. Existing methods are limited to reasoning over fixed initial inputs and lack the capacity to actively acquire further visual evidence, which is often essential for resolving complex references in long or intricate videos. To address this, we propose \textbf{VideoSEG-O3}, the first multi-turn reinforcement learning framework for RVOS that emulates the human \textit{``coarse-to-fine''} cognitive process. It employs a \textit{multi-turn temporal-spatial chain-of-thought} to capture fine-grained details by iteratively pinpointing critical intervals and keyframes. Additionally, to enable the policy to perceive segmentation quality beyond mere text probability of \texttt{[SEG]} during the RL stage, we introduce \textit{SEG-aware logit calibration}, which integrates pixel-wise segmentation feedback directly into the token-level logits. Furthermore, we design a \textit{decoupled thinking trace} to hierarchically decompose the reasoning process into temporal, spatial, and linguistic dimensions, and construct \textbf{VTS-CoT}, a specialized cold-start dataset featuring comprehensive reasoning trajectories. The code and models will be released at https://github.com/Dmmm1997/VideoSEG-O3.

Abstract

Explore similar work

VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation

MoVISA: Multi-Token Reasoning for Video Object Segmentation

QueenVIS: Rethinking Image-Only Training for Video Instance Segmentation via Query Enrichment