Learning to Reason with Persistent Object States for Video Instance Segmentation

Yongxue Xu; Boxue Yang; Ziqian Liu; Shaoqiu Zhang; Rui Qian; Haopeng Chen

cs.CVSep 28, 2026

Learning to Reason with Persistent Object States for Video Instance Segmentation

Authors: Yongxue Xu, Boxue Yang, Ziqian Liu, Shaoqiu Zhang, Rui Qian, Haopeng Chen

Organizations: Shanghai Jiao Tong University · Sun Yat-sen University · Fudan University

Abstract

Video segmentation models maintain object identities by carrying instance information across frames. Under prolonged occlusion, reappearance, or interactions between similar instances, however, an unreliable update can overwrite a valid history and cause persistent identity drift. We introduce POSReasoner, a trainable, plug-and-play framework that explicitly decides when an observation should change an object's state. Each persistent state records identity, confidence, and absence history. A sparse state-observation graph supports Propose-Verify reasoning: provisional associations are revisited using object history, predicted presence, and competition among identities. The verified decisions determine whether to retain, update, reactivate, or suppress each state, while a learned gate controls the evidence written back to memory. Only verified transitions update the persistent state used in subsequent frames. POSReasoner uses standard video annotations and keeps the base model frozen, enabling integration with diverse VOS and VIS architectures. Experiments across long-term VOS and VIS benchmarks show consistent improvements over strong baselines, with the largest gains under occlusion and object reappearance.

Figures & tables

Figure 1: POSReasoner preserves identity under crowding and similar-instance interactions (a), and reactivates returning objects after long gaps or dynamic motion (b). Yellow denotes references; teal denotes predictions.

Figure 2: From association to persistent-state reasoning. (a) Frame-local association leaves ambiguous matches unresolved across time. (b) Direct write-back can propagate one unreliable observation through later states. (c) POSReasoner keeps identity-indexed states separate from host memory, verifies competing hypotheses, and applies an explicit state transition.

Figure 3: Overview of POSReasoner. (a) Frozen-host observations are paired with persistent object states. (b) Propose–Verify reasoning refines associations and actions using identity history and object competition. The shaded matrices associate identities with observations or null. (c) Verified decisions gate a single state update. Internal values are schematic; DAVIS frames with ground-truth masks illustrate the interface, not model predictions.

Backbone	Method	YTVIS 2019			YTVIS 2021			YTVIS 2022
Backbone	Method	AP	AP50	AP75	AP	AP50	AP75	AP	AP50	AP75
R50	GenVIS ( Heo et al., 2023 )	50.0	71.5	54.6	47.1	67.5	51.5	37.5	61.6	41.5
	CTVIS ( Ying et al., 2023 )	51.8	74.1	55.2	49.5	72.5	53.6	45.4	67.5	48.9
	+ POSReasoner	52.8	75.2	56.5	50.6	73.7	55.1	47.5	69.7	51.4
	DVIS++ ( Zhang et al., 2023b )	55.7	80.8	59.8	50.4	71.4	55.2	46.3	66.6	50.5
	+ POSReasoner	56.0	80.6	60.9	51.1	72.6	55.8	47.0	68.2	51.0

Table 1: YouTube-VIS validation. Published methods provide context; each highlighted row adds POSReasoner to the host immediately above. Bold and underline mark the best and second-best result within each backbone.

Figure 4: Qualitative comparison under occlusion, identity ambiguity, and object reappearance.

ResNet-50
Method	$\Delta$ AP	AP	AP50	AP75
CTVIS ( Ying et al., 2023 )	–	$34.6$	$59.9$	$33.8$
+ POSReasoner	+2.7	$37.3$	$63.2$	$37.0$
DVIS++ ( Zhang et al., 2023b )	–	$36.8$	$61.3$	$37.5$
+ POSReasoner	+2.1	38.9	$64.3$	39.4
DAQ ( Zhou et al., 2024 )	–	$38.3$	65.2	$37.3$

Table 2: OVIS validation. Highlighted rows add POSReasoner to the matched host;

\Delta

AP is relative to that host. Bold and underline mark the best and second-best results within each backbone.

Variant	State	React.	Cross	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
LVOS v1 cumulative components
SAM3 host ( Carion et al., 2025 )	✗	✗	✗	79.5	90.6	85.1
+ state reasoning	$\checkmark$	✗	✗	82.1	92.4	87.2
+ reactivation	$\checkmark$	$\checkmark$	✗	83.3	93.6	88.4
Full POSReasoner	$\checkmark$	$\checkmark$	$\checkmark$	83.4	93.7	88.6
LVOS v2 transfer

Table 3: Long-term VOS and cumulative component analysis with frozen SAM3 predictions.

Figure 5: LVOS v1: persistent state preserves identity through absence (top); reactivation reconnects it on return (bottom).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	AP	$\Delta$ AP	AP50	AP75
Frozen host	34.63	–	59.94	33.80
Host cues only	34.82	+0.18	60.21	34.09
Shuffled state	34.67	+0.04	59.95	33.94
State, no graph	36.55	+1.92	62.88	35.61
State, one step	36.13	+1.50	62.01	35.27
State, shared two-step	36.97	+2.33	63.00	36.17

Appendix

Table 4: Causal controls on OVIS with the same frozen candidates and training protocol.

Candidates	Params	Latency	Memory
10	0.90M	5.80 ms	12.71 MB
20	0.90M	5.66 ms	12.99 MB
40	0.90M	5.64 ms	14.03 MB

Appendix

Table 5: Five-member VIS reasoner efficiency on one A100, batch size 1.

Dataset	Gap	Events	Videos	Host $\mathcal{J}$	POSR $\mathcal{J}$	$\Delta\mathcal{J}$ [95% CI]	H/T/D
LVOS v1	5–10	45	16	70.6	74.9	$+4.3$ [ $-2.3$ , $16.3$ ]	22/9/14
	11–30	34	15	51.1	64.2	$+13.1$ [ $-3.1$ , $29.1$ ]	20/6/8
	31–100	14	8	72.2	78.0	$+5.9$ [ $-2.7$ , $19.1$ ]	7/3/4
	$>100$	6	4	23.6	30.6	$+7.1$ [ $-10.1$ , $46.9$ ]	1/2/3
LVOS v2	5–10	114	42	66.9	65.8	$-1.1$ [ $-7.7$ , $5.3$ ]	53/17/44
	11–30	83	39	56.6	61.1	$+4.5$ [ $-2.3$ , $11.6$ ]	38/15/30

Appendix

Table 6: Post-reappearance results by absence duration, with paired differences and 95% bootstrap confidence intervals.

Method	Backbone	AP	AP50	AP75
GenVIS ( Heo et al., 2023 )	ResNet-50	35.8	60.8	36.2
CTVIS ( Ying et al., 2023 )	ResNet-50	35.5	60.8	34.9
DVIS++ ( Zhang et al., 2023b )	ResNet-50	37.2	62.8	37.3
DAQ ( Zhou et al., 2024 )	ResNet-50	38.7	65.5	37.6
CTVIS	ResNet-50	34.6	59.9	33.8
CTVIS + POSReasoner	ResNet-50	37.3	63.2	37.0

Appendix

Table 7: Complete OVIS evaluation across three backbones. Cited rows are published results; each uncited host is paired with POSReasoner under the same frozen host and candidate set.

Query-based Association
Method	Backbone	YTVIS 2019			YTVIS 2021			YTVIS 2022
Method	Backbone	AP	AP50	AP75	AP	AP50	AP75	AP	AP50	AP75
MinVIS ( Huang et al., 2022 )	R50	47.4	69.0	52.1	44.2	66.0	48.1	23.3	47.9	19.3
VITA ∗ ( Heo et al., 2022 )	R50	49.8	72.6	54.5	45.7	67.4	49.5	32.6	53.9	39.3
GenVIS ( Heo et al., 2023 )	R50	50.0	71.5	54.6	47.1	67.5	51.5	37.5	61.6	41.5
Memory and Decoupled Tracking

Appendix

Table 8: Complete YouTube-VIS plug-in evaluation; 2022 baselines follow LOMM ( Lee et al., 2025a ) . Each POSReasoner row retains its frozen host; ∗ denotes offline refinement and

\dagger

provisional results.

Explore similar work

Jun 5, 2026cs.CV

VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation

Reasoning Video Object Segmentation (RVOS) demands a sophisticated integration of temporal dynamics, spatial details, and linguistic reasoning to achieve precise pixel-level localization. Existing methods are limited to reasoning over fixed initial inputs and lack the capacity to actively acquire further visual evidence, which is often essential for resolving complex references in long or intricate videos. To address this, we propose \textbf{VideoSEG-O3}, the first multi-turn reinforcement learning framework for RVOS that emulates the human \textit{``coarse-to-fine''} cognitive process. It employs a \textit{multi-turn temporal-spatial chain-of-thought} to capture fine-grained details by iteratively pinpointing critical intervals and keyframes. Additionally, to enable the policy to perceive segmentation quality beyond mere text probability of \texttt{[SEG]} during the RL stage, we introduce \textit{SEG-aware logit calibration}, which integrates pixel-wise segmentation feedback directly into the token-level logits. Furthermore, we design a \textit{decoupled thinking trace} to hierarchically decompose the reasoning process into temporal, spatial, and linguistic dimensions, and construct \textbf{VTS-CoT}, a specialized cold-start dataset featuring comprehensive reasoning trajectories. The code and models will be released at https://github.com/Dmmm1997/VideoSEG-O3.

Ming Dai, Sen Yang, Boqiang Duan +4

School of Automation, Southeast University, Nanjing, China · Baidu Inc, Beijing, China · College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, China

Video Object Segmentation Chain-of-Thought Reasoning

Ask PDF

Sep 24, 2026cs.CV

MoVISA: Multi-Token Reasoning for Video Object Segmentation

Recent advances in video object segmentation with Multimodal Large Language Model (MLLM) reasoning have demonstrated the effectiveness of using a single textual token, such as SEG, to predict segmentation masks across images and videos. However, we observe that this single-token strategy lacks the granularity required to precisely localize multiple objects across time in video segmentation tasks. To address this limitation, we develop Multi-Token Reasoning for Video Object Segmentation, or MoVISA. MoVISA uses multiple segmentation tokens, such as SEG0 and SEG1, to represent an object across different frames. This design enables more fine-grained alignment between language prompts and spatio-temporal mask predictions, improving both performance and interpretability. On the challenging MeViS, DAVIS17, ReVOS, and Ref-Youtube-VOS benchmarks, our model achieves a 13.2 percent J and F improvement on MeViS and an 8.4 percent J and F improvement on ReVOS. Code and models will be released.

Ruining Zhao, Ho Kei Cheng, Alexander G Schwing

University of Illinois Urbana-Champaign

Video Object Segmentation Multimodal Large Language Models

Ask PDF

Jul 27, 2026cs.CV

QueenVIS: Rethinking Image-Only Training for Video Instance Segmentation via Query Enrichment

Video instance segmentation (VIS) requires models to detect, segment, and track object identities across frames, and most methods enforce temporal consistency through video-level supervision. Image-only training approaches, with MinVIS as one prominent example, have challenged this assumption, reaching competitive VIS without video training by treating frames as independent images and associating instances only at inference. The field has nonetheless moved toward ever more elaborate video-trained trackers, which depend on costly identity-consistent annotations, leaving the image-only direction under-explored. A diagnostic analysis identifies object query quality as the bottleneck: queries trained only to localize objects within a frame drift apart across frames and destabilize association. QueenVIS introduces a query-centric framework for strengthening image-trained VIS. During single-frame training, we enrich Mask2Former queries with two auxiliary heads: a feature-prediction loss that aligns each query with the pooled backbone descriptor of its instance, and a center-prediction loss that injects spatial structure. Both heads are discarded at inference, adding zero parameters, and temporal identity is maintained by a training-free query-propagation and memory-bank scheme. On YouTube-VIS and OVIS with a ResNet-50 backbone, QueenVIS improves over MinVIS, up to +6.7 AP on YouTube-VIS, +4.8 AP on OVIS, and +10.3 AP on the long-sequence YouTube-VIS split. QueenVIS achieves 50.9 AP on YouTube-VIS and remains competitive with recent video-supervised state-of-the-art, without processing a single video clip during training. Our findings suggest that strengthening the discriminative power and temporal stability of object queries is an important, underexplored axis for VIS. Code and models: https://github.com/ArianKheir/QueenVIS

Arian Kheirandish, Fardin Ayar, Ehsan Javanmardi +2

Amirkabir University of Technology (AUT), Tehran, Iran · The University of Tokyo, Tokyo, Japan

Video Object Segmentation Long Videos Queries

Ask PDF