RS-SSM: Refining Forgotten Specifics in State Space Model for Video Semantic Segmentation
Authors: Kai Zhu, Zhenyu Cui, Zehua Zang, Jiahuan Zhou
Organizations: Wangxuan Institute of Computer Technology, Peking University · Tsinghua University · Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences
Recently, state space models have demonstrated efficient video segmentation through linear-complexity state space compression. However, Video Semantic Segmentation (VSS) requires pixel-level spatiotemporal modeling capabilities to maintain temporal consistency in segmentation of semantic objects. While state space models can preserve common semantic information during state space compression, the fixed-size state space inevitably forgets specific information, which limits the models' capability for pixel-level segmentation. To tackle the above issue, we proposed a Refining Specifics State Space Model approach (RS-SSM) for video semantic segmentation, which performs complementary refining of forgotten spatiotemporal specifics. Specifically, a Channel-wise Amplitude Perceptron (CwAP) is designed to extract and align the distribution characteristics of specific information in the state space. Besides, a Forgetting Gate Information Refiner (FGIR) is proposed to adaptively invert and refine the forgetting gate matrix in the state space model based on the specific information distribution. Consequently, our RS-SSM leverages the inverted forgetting gate to complementarily refine the specific information forgotten during state space compression, thereby enhancing the model's capability for spatiotemporal pixel-level segmentation. Extensive experiments on four VSS benchmarks demonstrate that our RS-SSM achieves state-of-the-art performance while maintaining high computational efficiency. The code is available at https://github.com/zhoujiahuan1991/CVPR2026-RS-SSM.
Figures & tables
Figure 1: Existing SSM-based VSS methods lose spatiotemporal specifics when perform state space compression, limiting the model’s segmentation accuracy. In contrast, our proposed RS-SSM method guides the model to focus on forgotten spatiotemporal specifics, thereby improving the pixel-level semantic segmentation performance.
Figure 2: The pipeline of our RS-SSM. Following previous work [ 22 , 61 ] , we use an image encoder to extract feature maps for each frame. After linear projection, we extract spectrum features through the CwAP module to quantify the channel distribution of specific information. Then our FGIR module adaptively inverts and refines the forgetting gate from the SSM θ2 , thereby encouraging SSM θ1 to perform complementary refining of forgotten spatiotemporal specifics.
Figure 3: Visualization of the updating gate Bd as mentioned in Eq 2 . Influenced by the forgetting gate, the updating gate reflects the amount of new information introduced at each time step. The bottom row visualization reveals that the vanilla SSM θ2 suffers from information loss of specifics during the state space compression. Conversely, as illustrated in the top row, the SSM θ1 effectively refines these forgotten specifics after inverting and refining the forgetting gate.
Table 1: The comparison with existing methods on the VSPW [ 37 ] , NYUv2 [ 52 ] and CamVid [ 1 ] dataset. Our RS-SSM achieves state-of-the-art performance while maintaining high efficiency. The best and second-best results are highlighted in RED and BLUE , respectively. We calculate GFLOPs and FPS with an input resolution of 480 × 853.
Figure 4: Visualization of segmentation results on VSPW dataset [ 37 ] . Compared to the existing SSM-based method TV3S [ 22 ] , our RS-SSM produces more accurate and detailed segmentation results by effectively refining specific information in videos.
\rectanglecolor gray!209-19-6 Methods
Backbones
mIoU ↑
GFLOPs ↓
Params(M) ↓
FPS ↑
SegFormer [ 61 ]
MiT-B0
71.9
54.6
3.7
58.5
CFFM [ 53 ]
MiT-B0
74.0
80.7
4.6
15.8
MRCFA [ 55 ]
MiT-B0
72.8
77.5
4.2
16.6
SegFormer [ 61 ]
MiT-B1
74.1
106.4
13.7
46.8
CFFM++ [ 54 ]
MiT-B1
75.7
169.8
15.9
20.4
MRCFA [ 55 ]
MiT-B1
75.1
145.0
14.9
13.0
Table 2: The comparison with existing methods on the Cityscapes [ 7 ] dataset, using an input size of 512 × 1024.
\rectanglecolor gray!206-16-6 \Block 2-1Methods
\Block 1-5mIoU ↑
MiT-B1
MiT-B2
MiT-B5
Swin-T
Swin-S
V-SSM
38.8
45.1
48.5
43.3
49.8
Bi-V-SSM
39.6
45.4
49.0
43.7
50.8
No-CwAP
40.7
46.0
50.2
44.9
51.6
RS-SSM
41.7
46.8
51.6
45.7
52.2
Table 3: Ablation study on the influence of different components in our RS-SSM. V-SSM represents only using one vanilla State Space Model path, Bi-V-SSM represents using two-way vanilla State Space Model paths, and No-CwAP using our RS-SSM with a simply inverted forgetting gate as mentioned in Eq 16 .
Fully supervised Video Semantic Segmentation (VSS) relies heavily on densely annotated video data, limiting practical applicability. Alternatively, applying pre-trained Image Semantic Segmentation (ISS) models frame-by-frame avoids annotation costs but ignores crucial temporal coherence. Recent foundation models such as SAM2 enable high-quality mask propagation yet remain impractical for direct VSS due to limited semantic understanding and computational overhead. In this paper, we propose DiTTA (Distillation-assisted Test-Time Adaptation), a novel framework that converts an ISS model into a temporally-aware VSS model through efficient test-time adaptation (TTA), without annotated videos. DiTTA distills SAM2's temporal segmentation knowledge into the ISS model during a brief, single-pass initialization phase, complemented by a lightweight temporal fusion module to aggregate cross-frame context. Crucially, DiTTA achieves robust generalization even when adapting with highly limited partial video snippets (e.g., initial 10%), significantly outperforming zero-shot refinement approaches that repeatedly invoke SAM2 during inference. Extensive experiments on VSPW and Cityscapes demonstrate DiTTA's effectiveness, achieving competitive or superior performance relative to fully-supervised VSS methods, thus providing a practical and annotation-free solution for real-world VSS tasks.
Video Reasoning Segmentation (VRS) aims to segment target objects in videos based on implicit instructions that convey human intent and temporal logic. Existing MLLM-based methods predict masks with a [SEG] token after selecting frames via simple sampling or an auxiliary MLLM, where limited supervision and frame-language similarity rules often yield narrow-scope keyframe choices that weaken holistic temporal understanding and lead to brittle localization in complex multi-object scenes. To address these issues, we introduce RCoT-Seg, a video-of-thought framework that factorizes VRS into temporal video reasoning (TVR) and keyframe target perception (KTP), explicitly separating temporal reasoning from spatial perception. Specifically, in the TVR stage, an agentic keyframe selection module, initialized with a curated CoT-start corpus and refined by GRPO under task-aligned rewards, is proposed to generate and reselect the keyframe through self-evaluation, strengthening moment localization and temporal reasoning. In the KTP stage, RCoT-Seg performs high-resolution segmentation on the selected frame and propagates masks with SAM2-based methods across the sequence, replacing heuristic sampling and external selectors while improving spatial precision and inter-frame consistency. Extensive experimental results demonstrate that the proposed RCoT-Seg achieves favorable performance against the state-of-the-art methods. The code and models will be publicly released at https://github.com/Victor-wjw/RCoT-Seg.
Junwei Wen, Deshui Miao, Guangming Lu +2
Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · Pazhou Lab (Huangpu)
Referring Video Object Segmentation (RVOS) aims to segment target objects in videos based on natural language descriptions. However, fixed keyframe-based approaches that couple a vision language model with a separate propagation module often fail to capture rapidly changing spatiotemporal dynamics and to handle queries requiring multi-step reasoning, leading to sharp performance drops on motion-intensive and reasoning-oriented videos beyond static RVOS benchmarks. To address these limitations, we propose VIRST (Video-Instructed Reasoning Assistant for Spatio-Temporal Segmentation), an end-to-end framework that unifies global video reasoning and pixel-level mask prediction within a single model. VIRST bridges semantic and segmentation representations through the Spatio-Temporal Fusion (STF), which fuses segmentation-aware video features into the vision-language backbone, and employs the Temporal Dynamic Anchor Updater to maintain temporally adjacent anchor frames that provide stable temporal cues under large motion, occlusion, and reappearance. This unified design achieves state-of-the-art results across diverse RVOS benchmarks under realistic and challenging conditions, demonstrating strong generalization to both referring and reasoning oriented settings. The code and checkpoints are available at https://github.com/AIDASLab/VIRST.
Jihwan Hong, Jaeyoung Do
AIDAS Laboratory, 1IPAI, Seoul National University · AIDAS Laboratory, 2ECE, Seoul National University