In this paper, we introduce Predicate-State Video Reasoning Segmentation (PS-VRS), where a query defines a dynamic predicate P(o,t) over objects and time rather than a fixed object identity. Existing methods typically treat the target as permanently active once identified, leading to over-emission of masks during vacant intervals and failure to handle target switching. To address this, we construct PS-Bench, a large-scale benchmark with 2,077 videos and 16,250 queries annotated with explicit active/vacant predicate states, temporal intervals, and pixel masks. We further propose Predicate-State Grounding (PSG), a State-Bind-Emit pipeline that: (1) predicts [ACTIVE]/[VACANT] state tokens; (2) aligns them to video frames via dense cross-modal similarity; and (3) emits masks via SAM2 only for active states. Experiments show that PSG outperforms existing baselines by +9.32 J&F on PS-Bench and significantly reduces false-positive mask emissions. Our results demonstrate the effectiveness of explicit predicate-state modeling for video reasoning segmentation with dynamic visual witnesses.
Figures & tables
Figure 1: Task formulation comparison. Referring Video Object Segmentation segments explicitly referred targets, while Video Reasoning Segmentation introduces implicit reasoning queries but typically lacks explicit target switching and temporal predicate-state boundaries. PS-VRS treats the query as a predicate over objects and time, requiring reasoning, target switching, active/vacant temporal grounding, and pixel-level masks for active segments.
Benchmark
Videos
Dur./Vid.
Queries
Masks
Avg. Segs. (A/V)
Ref-YouTube-VOS
3,978
5.45s
15,009
131k
1 †
MeViS
2,006
13.16s
33,072
443k
1 †
Ref-DAVIS17
90
2.87s
1,544
13.5k
1 †
GroundMore
1.7k
9.61s
7.6k
249.5k
1 †
PS-Bench
2,077
45.81s
16,250
1,569k
3.82 (2.18/1.64)
Table 1: Statistics of representative language-guided video segmentation benchmarks. A/V denotes active/vacant predicate-state segments. † Existing datasets do not explicitly annotate active/vacant predicate states and are treated as single active segments for comparison.
Figure 2: Overview of the Predicate-State Grounding (PSG) Segmentation architecture. PSG decouples semantic validity from mask generation via a strict State-Bind-Emit pipeline. (1) State: A Multimodal Large Language Model (MLLM) infers the dynamic predicate states by generating discrete structural reasoning tokens ( [ACTIVE] , [VACANT] , and [SEG] ). (2) Bind: The Span-Frame Projecting module performs dense cross-modal alignment, binding these abstract state tokens to continuous video frames to establish frame-level temporal boundaries. (3) Emit: The [SEG] token acts as a semantic prompt for SAM to decode masks exclusively during active spans. Mask decoding is bypassed during predicted vacant intervals, substantially reducing invalid mask emission.
Method
Params
Spatial
Temporal
J
F
J&F
mIoU
VT
TS
All
VT
TS
All
VT
TS
All
VT
TS
All
Predicate-State VRS Methods (Adapted from Large-scale MLLM)
Gemini-3.1-Pro
N/A
56.78
54.66
55.25
58.06
55.50
56.21
57.42
55.08
55.73
47.62
43.68
44.49
Qwen3-VL-Max
235B
60.39
51.08
53.71
63.10
53.71
56.37
61.75
52.39
55.05
56.38
51.86
52.81
Kimi-K2.5
1T
60.96
55.04
56.68
63.78
56.06
58.20
62.37
55.55
57.44
57.21
49.84
51.30
Table 2: Quantitative comparison on PS-Bench . The baseline methods are grouped into adapted large-scale MLLMs, previous VRS models, and our baseline comparison. Our proposed PSG establishes a new state-of-the-art for predicate-state VRS. It outperforms previous specialized methods by a large margin and demonstrates highly competitive spatial and temporal grounding against massive MLLMs using only 4B parameters. VT and TS denote the Visibility Transition and Target Switching subsets, respectively. "–" indicates the inability of previous VRS models to predict explicit temporal state boundaries, making temporal evaluation inapplicable.
Method
J
F
J&F
VideoGLaMM
42.1
48.2
45.2
VideoLISA
41.3
47.6
44.4
VISA
41.8
47.1
44.5
CoT-RVS
42.7
49.1
45.9
ViLLa
46.6
52.3
49.4
VRS-HQ
48.0
53.7
50.9
Table 3: Quantitative comparison on MeViS . † denotes our model trained exclusively on PS-Bench and directly evaluated on MeViS without in-domain supervision. With the complete training mixture, PSG achieves the best overall performance.
Method
J
F
J&F
with state
61.26
62.71
61.99
w/o vacant
55.08
56.87
55.97
w/o state
30.99
32.60
31.79
Table 4: Ablation study on predicate-state modeling. Removing explicit state modeling causes a large performance drop, while replacing [VACANT] with thresholding also degrades performance.
Figure 3: Qualitative comparison on dynamic predicate-state transitions. For the query about the cleaning implement currently being used, the valid visual witness changes over time and is absent in the initial frames. PSG suppresses mask emission during the vacant state and rebinds the active state to the query-valid witness when the used implement changes. In contrast, existing baselines often segment visually plausible but query-invalid objects, or continue emitting masks across invalid states.
Method
False Active ↓
False Vacant ↓
VISA
49.23
1.13
VideoLISA
53.67
0.12
VRS-HQ
26.00
10.42
GLUS
31.44
5.09
Ours
19.70
12.09
Table 5: State errors on PS-Bench . False Active denotes the percentage of ground-truth vacant frames predicted as active, while False Vacant denotes the percentage of ground-truth active frames predicted as vacant.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Explicit baseline prompt. System prompt used to adapt generalist MLLMs to predicate-state temporal segmentation with explicit active/vacant states, target-set transitions, and bounding-box localization.
Method
Relative Runtime ↓
VideoLISA
4.0 ×
VRS-HQ
4.0 ×
GLUS
9.0 ×
Ours
1.0 ×
Appendix
Table 6: Normalized end-to-end inference time on the PS-Bench test set, with the runtime of PSG normalized to 1.0 . All methods are evaluated under the same hardware and implementation protocol.
Category
Distribution
Data split
Train / Val. / Test = 7:2:1
Source videos
CoVLA: 817; ActivityNet: 906; VDC: 354
Predicate state
Active: 46.6%; Vacant: 53.4%
Transition type
VT: 22.96%; TS: 77.04%
Appendix
Table 7: Detailed statistics of PS-Bench . The active/vacant distribution is computed over query-conditioned frame annotations, while the VT/TS distribution is manually computed over the test split.
Figure 5: Distribution of pairwise bounding-box IoUs between the NVIDIA and Ascend deployments of Qwen3-VL-8B-Instruct. IoU is computed over the 559 annotations for which both deployments select the same candidate frame. The median IoU is 0.979 .
Figure 6: Qualitative cross-platform comparison of bounding-box annotations generated by Qwen3-VL-8B-Instruct. Each row corresponds to the same query and candidate key frame, with the NVIDIA prediction shown on the left in red and the Ascend prediction shown on the right in cyan. Pairwise IoU is reported in the upper-left corner. The examples include both consistent and inconsistent predictions across the two deployment stacks.
Video Reasoning Segmentation (VRS) aims to infer and segment target objects specified by reasoning queries in videos. Existing methods typically first infer the target and generate a few segmentation tokens for it, which are then fed into a mask decoder for mask prediction. However, such segmentation tokens tend to be both semantically monotonous and spatially ambiguous, making them largely insufficient to represent the target to guide mask prediction. In this paper, we propose an explicit visual object grounding framework for VRS. Specifically, we first propose a spatial-aware prompt generation and refinement scheme, in which we query a multimodal large language model to generate explicit spatial prompts (i.e., bounding box and point set) for the target and then refine them through multi-step dialogue. Furthermore, we introduce a query diversification and alignment module to generate auxiliary queries that describe the same target from different perspectives. We then enforce cross-query consistency between their predicted spatial prompts to alleviate the training bias caused by query limitations. Finally, we select several reliable frames using a prompt-based keyframe selection strategy to support mask decoding and propagation. Extensive experiments on eight datasets demonstrate that our method significantly outperforms state-of-the-art methods. The code is available at https://github.com/viscom-tongji/EVOG.
Jun Huang, Junyang Zhang, Zijie Yue +3
College of Electronic and Information Engineering, Tongji University · Shanghai Zhenhua Heavy Industries Co., Ltd. · College of Computer Science, Wuhan University +1
Video segmentation models maintain object identities by carrying instance information across frames. Under prolonged occlusion, reappearance, or interactions between similar instances, however, an unreliable update can overwrite a valid history and cause persistent identity drift. We introduce POSReasoner, a trainable, plug-and-play framework that explicitly decides when an observation should change an object's state. Each persistent state records identity, confidence, and absence history. A sparse state-observation graph supports Propose-Verify reasoning: provisional associations are revisited using object history, predicted presence, and competition among identities. The verified decisions determine whether to retain, update, reactivate, or suppress each state, while a learned gate controls the evidence written back to memory. Only verified transitions update the persistent state used in subsequent frames. POSReasoner uses standard video annotations and keeps the base model frozen, enabling integration with diverse VOS and VIS architectures. Experiments across long-term VOS and VIS benchmarks show consistent improvements over strong baselines, with the largest gains under occlusion and object reappearance.
Yongxue Xu, Boxue Yang, Ziqian Liu +3
Shanghai Jiao Tong University · Sun Yat-sen University · Fudan University
Video Reasoning Segmentation (VRS) requires a model to identify an implicitly described target and predict its pixel-level mask trajectory, while resolving the spatial relations and temporal events that determine the answer. Text-only Chain-of-Thought (CoT) can explain a target choice without making the underlying visual references explicit, leaving a video reference gap between linguistic inference and the evidence distributed across frames. We introduce Temporal Chain of Video Primitives (TCoVP), a spatiotemporally grounded form of Video CoT that organizes query-relevant observations into a chronological reasoning chain. Each primitive associates an entity reference with temporal support and timestamped spatial evidence, and the chain interleaves these references with language to guide reasoning about which entity is relevant, when the queried event holds, and where its evidence appears. The selected target primitives then provide box prompts and temporal support for video mask prediction. We construct TCoVP-SFT-15k and TCoVP-RL-13k to learn this representation through supervised fine-tuning and chain-aligned reinforcement learning. Extensive experiments across five VRS benchmarks and three model backbones demonstrate the strong performance of TCoVP, highlighting the effectiveness of organizing spatial and temporal evidence within the reasoning process. All models and code will be publicly released. Project page: https://victor-wjw.github.io/TCoVP/
Junwei Wen, Deshui Miao, Guangming Lu +3
Harbin Institute of Technology, Shenzhen · University of California at Merced · Pengcheng Laboratory +1