RS-SSM: Refining Forgotten Specifics in State Space Model for Video Semantic Segmentation

Kai Zhu; Zhenyu Cui; Zehua Zang; Jiahuan Zhou

cs.CVMar 25, 2026

RS-SSM: Refining Forgotten Specifics in State Space Model for Video Semantic Segmentation

Authors: Kai Zhu, Zhenyu Cui, Zehua Zang, Jiahuan Zhou

Organizations: Wangxuan Institute of Computer Technology, Peking University · Tsinghua University · Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences

Abstract

Recently, state space models have demonstrated efficient video segmentation through linear-complexity state space compression. However, Video Semantic Segmentation (VSS) requires pixel-level spatiotemporal modeling capabilities to maintain temporal consistency in segmentation of semantic objects. While state space models can preserve common semantic information during state space compression, the fixed-size state space inevitably forgets specific information, which limits the models' capability for pixel-level segmentation. To tackle the above issue, we proposed a Refining Specifics State Space Model approach (RS-SSM) for video semantic segmentation, which performs complementary refining of forgotten spatiotemporal specifics. Specifically, a Channel-wise Amplitude Perceptron (CwAP) is designed to extract and align the distribution characteristics of specific information in the state space. Besides, a Forgetting Gate Information Refiner (FGIR) is proposed to adaptively invert and refine the forgetting gate matrix in the state space model based on the specific information distribution. Consequently, our RS-SSM leverages the inverted forgetting gate to complementarily refine the specific information forgotten during state space compression, thereby enhancing the model's capability for spatiotemporal pixel-level segmentation. Extensive experiments on four VSS benchmarks demonstrate that our RS-SSM achieves state-of-the-art performance while maintaining high computational efficiency. The code is available at https://github.com/zhoujiahuan1991/CVPR2026-RS-SSM.

Figures & tables

Figure 1: Existing SSM-based VSS methods lose spatiotemporal specifics when perform state space compression, limiting the model’s segmentation accuracy. In contrast, our proposed RS-SSM method guides the model to focus on forgotten spatiotemporal specifics, thereby improving the pixel-level semantic segmentation performance.

Figure 2: The pipeline of our RS-SSM. Following previous work [ 22 , 61 ] , we use an image encoder to extract feature maps for each frame. After linear projection, we extract spectrum features through the CwAP module to quantify the channel distribution of specific information. Then our FGIR module adaptively inverts and refines the forgetting gate from the SSM $\theta_{2}$ , thereby encouraging SSM $\theta_{1}$ to perform complementary refining of forgotten spatiotemporal specifics.

Figure 3: Visualization of the updating gate $\mathbf{\overline{B}}_{d}$ as mentioned in Eq 2 . Influenced by the forgetting gate, the updating gate reflects the amount of new information introduced at each time step. The bottom row visualization reveals that the vanilla SSM $\theta_{2}$ suffers from information loss of specifics during the state space compression. Conversely, as illustrated in the top row, the SSM $\theta_{1}$ effectively refines these forgotten specifics after inverting and refining the forgetting gate.

\rectanglecolor gray!207-17-8 \rectanglecolor gray!2012-112-8 \rectanglecolor gray!2021-121-8 \rectanglecolor gray!2025-125-8 \rectanglecolor gray!2029-129-8 \Block 2-1Methods								\Block 2-1Backbones	\Block 1-3mIoU $\uparrow$	\Block 2-1GFLOPs $\downarrow$	\Block 2-1Params(M) $\downarrow$	\Block 2-1FPS $\uparrow$
		VSPW	NYUv2	CamVid
Segformer [ 61 ]	MiT-B1	36.5	37.6	57.9	26.6	13.8	58.7
CFFM++ [ 54 ]	MiT-B1	39.9	39.5	60.1	114.3	16.5	27.6
MRCFA [ 55 ]	MiT-B1	38.9	39.1	59.7	77.5	16.2	40.1
TV3S [ 22 ]	MiT-B1	40.0	39.7	60.8	36.9	17.3	24.7
RS-SSM (Ours)	MiT-B1	41.7	41.4	62.7	48.1	20.2	37.3

Table 1: The comparison with existing methods on the VSPW [ 37 ] , NYUv2 [ 52 ] and CamVid [ 1 ] dataset. Our RS-SSM achieves state-of-the-art performance while maintaining high efficiency. The best and second-best results are highlighted in RED and BLUE , respectively. We calculate GFLOPs and FPS with an input resolution of 480

\times

853.

Figure 4: Visualization of segmentation results on VSPW dataset [ 37 ] . Compared to the existing SSM-based method TV3S [ 22 ] , our RS-SSM produces more accurate and detailed segmentation results by effectively refining specific information in videos.

\rectanglecolor gray!209-19-6 Methods						Backbones	mIoU $\uparrow$	GFLOPs $\downarrow$	Params(M) $\downarrow$	FPS $\uparrow$
SegFormer [ 61 ]	MiT-B0	71.9	54.6	3.7	58.5
CFFM [ 53 ]	MiT-B0	74.0	80.7	4.6	15.8
MRCFA [ 55 ]	MiT-B0	72.8	77.5	4.2	16.6
SegFormer [ 61 ]	MiT-B1	74.1	106.4	13.7	46.8
CFFM++ [ 54 ]	MiT-B1	75.7	169.8	15.9	20.4
MRCFA [ 55 ]	MiT-B1	75.1	145.0	14.9	13.0

Table 2: The comparison with existing methods on the Cityscapes [ 7 ] dataset, using an input size of 512

\times

1024.

\rectanglecolor gray!206-16-6 \Block 2-1Methods						\Block 1-5mIoU $\uparrow$
	MiT-B1	MiT-B2	MiT-B5	Swin-T	Swin-S
V-SSM	38.8	45.1	48.5	43.3	49.8
Bi-V-SSM	39.6	45.4	49.0	43.7	50.8
No-CwAP	40.7	46.0	50.2	44.9	51.6
RS-SSM	41.7	46.8	51.6	45.7	52.2

Table 3: Ablation study on the influence of different components in our RS-SSM. V-SSM represents only using one vanilla State Space Model path, Bi-V-SSM represents using two-way vanilla State Space Model paths, and No-CwAP using our RS-SSM with a simply inverted forgetting gate as mentioned in Eq 16 .

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 414 papers

Bootstrapping Video Semantic Segmentation Model via Distillation-assisted Test-Time Adaptation
Apr 13, 2026Jihun Kim, Hoyong Kwon, Hyeokjun Kweon +1Semantic Segmentation Test-Time Adaptation
KAIST · Chung-Ang University
RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation
May 8, 2026Junwei Wen, Deshui Miao, Guangming Lu +2Video Object Segmentation Video Reasoning
Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · Pazhou Lab (Huangpu)
VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation
Mar 28, 2026Jihwan Hong, Jaeyoung DoVideo Object Segmentation Referring Video Object Segmentation
AIDAS Laboratory, 1IPAI, Seoul National University · AIDAS Laboratory, 2ECE, Seoul National University

Apr 13, 2026J/K move · Enter open · S save

Bootstrapping Video Semantic Segmentation Model via Distillation-assisted Test-Time Adaptation

Jihun Kim, Hoyong Kwon, Hyeokjun Kweon, Kuk-Jin Yoon

KAIST · Chung-Ang University

Semantic Segmentation Test-Time Adaptation

Open paper AskPDF arXiv

Fully supervised Video Semantic Segmentation (VSS) relies heavily on densely annotated video data, limiting practical applicability. Alternatively, applying pre-trained Image Semantic Segmentation (ISS) models frame-by-frame avoids annotation costs but ignores crucial temporal coherence. Recent foundation models such as SAM2 enable high-quality mask propagation yet remain impractical for direct VSS due to limited semantic understanding and computational overhead. In this paper, we propose DiTTA (Distillation-assisted Test-Time Adaptation), a novel framework that converts an ISS model into a temporally-aware VSS model through efficient test-time adaptation (TTA), without annotated videos. DiTTA distills SAM2's temporal segmentation knowledge into the ISS model during a brief, single-pass initialization phase, complemented by a lightweight temporal fusion module to aggregate cross-frame context. Crucially, DiTTA achieves robust generalization even when adapting with highly limited partial video snippets (e.g., initial 10%), significantly outperforming zero-shot refinement approaches that repeatedly invoke SAM2 during inference. Extensive experiments on VSPW and Cityscapes demonstrate DiTTA's effectiveness, achieving competitive or superior performance relative to fully-supervised VSS methods, thus providing a practical and annotation-free solution for real-world VSS tasks.

Abstract

Semantic Segmentation

Explore similar work

Bootstrapping Video Semantic Segmentation Model via Distillation-assisted Test-Time Adaptation

RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation

VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation