In this paper, we introduce Predicate-State Video Reasoning Segmentation (PS-VRS), where a query defines a dynamic predicate P(o,t) over objects and time rather than a fixed object identity. Existing methods typically treat the target as permanently active once identified, leading to over-emission of masks during vacant intervals and failure to handle target switching. To address this, we construct PS-Bench, a large-scale benchmark with 2,077 videos and 16,250 queries annotated with explicit active/vacant predicate states, temporal intervals, and pixel masks. We further propose Predicate-State Grounding (PSG), a State-Bind-Emit pipeline that: (1) predicts [ACTIVE]/[VACANT] state tokens; (2) aligns them to video frames via dense cross-modal similarity; and (3) emits masks via SAM2 only for active states. Experiments show that PSG outperforms existing baselines by +9.32 J&F on PS-Bench and significantly reduces false-positive mask emissions. Our results demonstrate the effectiveness of explicit predicate-state modeling for video reasoning segmentation with dynamic visual witnesses.
Figures & tables
Figure 1: Task formulation comparison. Referring Video Object Segmentation segments explicitly referred targets, while Video Reasoning Segmentation introduces implicit reasoning queries but typically lacks explicit target switching and temporal predicate-state boundaries. PS-VRS treats the query as a predicate over objects and time, requiring reasoning, target switching, active/vacant temporal grounding, and pixel-level masks for active segments.
Benchmark
Videos
Dur./Vid.
Queries
Masks
Avg. Segs. (A/V)
Ref-YouTube-VOS
3,978
5.45s
15,009
131k
1 †
MeViS
2,006
13.16s
33,072
443k
1 †
Ref-DAVIS17
90
2.87s
1,544
13.5k
1 †
GroundMore
1.7k
9.61s
7.6k
249.5k
1 †
PS-Bench
2,077
45.81s
16,250
1,569k
3.82 (2.18/1.64)
Table 1: Statistics of representative language-guided video segmentation benchmarks. A/V denotes active/vacant predicate-state segments. † Existing datasets do not explicitly annotate active/vacant predicate states and are treated as single active segments for comparison.
Figure 2: Overview of the Predicate-State Grounding (PSG) Segmentation architecture. PSG decouples semantic validity from mask generation via a strict State-Bind-Emit pipeline. (1) State: A Multimodal Large Language Model (MLLM) infers the dynamic predicate states by generating discrete structural reasoning tokens ( [ACTIVE] , [VACANT] , and [SEG] ). (2) Bind: The Span-Frame Projecting module performs dense cross-modal alignment, binding these abstract state tokens to continuous video frames to establish frame-level temporal boundaries. (3) Emit: The [SEG] token acts as a semantic prompt for SAM to decode masks exclusively during active spans. Mask decoding is bypassed during predicted vacant intervals, substantially reducing invalid mask emission.
Method
Params
Spatial
Temporal
J
F
J&F
mIoU
VT
TS
All
VT
TS
All
VT
TS
All
VT
TS
All
Predicate-State VRS Methods (Adapted from Large-scale MLLM)
Gemini-3.1-Pro
N/A
56.78
54.66
55.25
58.06
55.50
56.21
57.42
55.08
55.73
47.62
43.68
44.49
Qwen3-VL-Max
235B
60.39
51.08
53.71
63.10
53.71
56.37
61.75
52.39
55.05
56.38
51.86
52.81
Kimi-K2.5
1T
60.96
55.04
56.68
63.78
56.06
58.20
62.37
55.55
57.44
57.21
49.84
51.30
Table 2: Quantitative comparison on PS-Bench . The baseline methods are grouped into adapted large-scale MLLMs, previous VRS models, and our baseline comparison. Our proposed PSG establishes a new state-of-the-art for predicate-state VRS. It outperforms previous specialized methods by a large margin and demonstrates highly competitive spatial and temporal grounding against massive MLLMs using only 4B parameters. VT and TS denote the Visibility Transition and Target Switching subsets, respectively. "–" indicates the inability of previous VRS models to predict explicit temporal state boundaries, making temporal evaluation inapplicable.
Method
J
F
J&F
VideoGLaMM
42.1
48.2
45.2
VideoLISA
41.3
47.6
44.4
VISA
41.8
47.1
44.5
CoT-RVS
42.7
49.1
45.9
ViLLa
46.6
52.3
49.4
VRS-HQ
48.0
53.7
50.9
Table 3: Quantitative comparison on MeViS . † denotes our model trained exclusively on PS-Bench and directly evaluated on MeViS without in-domain supervision. With the complete training mixture, PSG achieves the best overall performance.
Method
J
F
J&F
with state
61.26
62.71
61.99
w/o vacant
55.08
56.87
55.97
w/o state
30.99
32.60
31.79
Table 4: Ablation study on predicate-state modeling. Removing explicit state modeling causes a large performance drop, while replacing [VACANT] with thresholding also degrades performance.
Figure 3: Qualitative comparison on dynamic predicate-state transitions. For the query about the cleaning implement currently being used, the valid visual witness changes over time and is absent in the initial frames. PSG suppresses mask emission during the vacant state and rebinds the active state to the query-valid witness when the used implement changes. In contrast, existing baselines often segment visually plausible but query-invalid objects, or continue emitting masks across invalid states.
Method
False Active ↓
False Vacant ↓
VISA
49.23
1.13
VideoLISA
53.67
0.12
VRS-HQ
26.00
10.42
GLUS
31.44
5.09
Ours
19.70
12.09
Table 5: State errors on PS-Bench . False Active denotes the percentage of ground-truth vacant frames predicted as active, while False Vacant denotes the percentage of ground-truth active frames predicted as vacant.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Explicit baseline prompt. System prompt used to adapt generalist MLLMs to predicate-state temporal segmentation with explicit active/vacant states, target-set transitions, and bounding-box localization.
Method
Relative Runtime ↓
VideoLISA
4.0 ×
VRS-HQ
4.0 ×
GLUS
9.0 ×
Ours
1.0 ×
Appendix
Table 6: Normalized end-to-end inference time on the PS-Bench test set, with the runtime of PSG normalized to 1.0 . All methods are evaluated under the same hardware and implementation protocol.
Category
Distribution
Data split
Train / Val. / Test = 7:2:1
Source videos
CoVLA: 817; ActivityNet: 906; VDC: 354
Predicate state
Active: 46.6%; Vacant: 53.4%
Transition type
VT: 22.96%; TS: 77.04%
Appendix
Table 7: Detailed statistics of PS-Bench . The active/vacant distribution is computed over query-conditioned frame annotations, while the VT/TS distribution is manually computed over the test split.
Figure 5: Distribution of pairwise bounding-box IoUs between the NVIDIA and Ascend deployments of Qwen3-VL-8B-Instruct. IoU is computed over the 559 annotations for which both deployments select the same candidate frame. The median IoU is 0.979 .
Figure 6: Qualitative cross-platform comparison of bounding-box annotations generated by Qwen3-VL-8B-Instruct. Each row corresponds to the same query and candidate key frame, with the NVIDIA prediction shown on the left in red and the Ascend prediction shown on the right in cyan. Pairwise IoU is reported in the upper-left corner. The examples include both consistent and inconsistent predictions across the two deployment stacks.
College of Electronic and Information Engineering, Tongji University · Shanghai Zhenhua Heavy Industries Co., Ltd. · College of Computer Science, Wuhan University +1