SF-Mamba: Rethinking State Space Model for Vision

Masakazu Yoshimura; Teruaki Hayashi; Yuki Hoshino; Wei-Yao Wang; Takeshi Ohashi

cs.CVMar 17, 2026

SF-Mamba: Rethinking State Space Model for Vision

Authors: Masakazu Yoshimura, Teruaki Hayashi, Yuki Hoshino, Wei-Yao Wang, Takeshi Ohashi

Organizations: Sony Group Corporation, Tokyo, Japan.

Abstract

The realm of Mamba for vision has been advanced in recent years to strike for the alternatives of Vision Transformers (ViTs) that suffer from the quadratic complexity. While the recurrent scanning mechanism of Mamba offers computational efficiency, it inherently limits non-causal interactions between image patches. Prior works have attempted to address this limitation through various multi-scan strategies; however, these approaches suffer from inefficiencies due to suboptimal scan designs and frequent data rearrangement. Moreover, Mamba exhibits relatively slow computational speed under short token lengths, commonly used in visual tasks. In pursuit of a truly efficient vision encoder, we rethink the scan operation for vision and the computational efficiency of Mamba. To this end, we propose SF-Mamba, a novel visual Mamba with two key proposals: auxiliary patch swapping for encoding bidirectional information flow under an unidirectional scan and batch folding with periodic state reset for advanced GPU parallelism. Extensive experiments on image classification, object detection, and instance and semantic segmentation consistently demonstrate that our proposed SF-Mamba significantly outperforms state-of-the-art baselines while improving throughput across different model sizes. We will release the source code after publication.

Figures & tables

Figure 1: Top-1 accuracy and throughput on ImageNet-1K classification. SF-Mamba offers superior accuracy–throughput trade-offs compared to state-of-the-art architectures.

Figure 2: Future-to-Past Information Routing via Auxiliary Token Swapping. The left figure illustrates why the commonly used multi-directional scan in visual Mamba fails to achieve high speed, while the right figure presents our proposed solution. We prepend/append learnable auxiliary tokens to the patch sequence $x^{\text{aux}}_{\text{head}}$ and $x^{\text{aux}}_{\text{tail}}$ . Within each MambaVision block, the causal selective scan aggregates sequence-wide context into the tail token $y^{\text{aux}}_{\text{tail}}$ . A lightweight, parameter-free Swap operation then moves this global summary to the sequence head, yielding $\tilde{X}$ for the next layer such that all patch states are conditioned on global context. It incurs negligible computational overhead while enabling effective global-context propagation across layers.

Panel of Figure 3 — Figure 3: Batch folding with periodic state reset. ( Left ) An input tensor of shape $[B,D,T]$ is reshaped into $[B_{1},D,(B_{2}\cdot T)]$ , concatenating $B_{2}$ short sequences into a longer one. This reshaping mixes hidden states across batches. ( Right ) To avoid information leakage, we reset the recurrence every $T$ steps. Since $h_{t}\leftarrow A_{t}h_{t-1}+B_{t}x_{t}$ , setting $A_{t}=0$ at boundaries is equivalent to re-initializing the hidden state. In contrast, $B_{t}$ (input projection) and $C_{t}$ (output projection) operate locally and therefore remain unchanged.

Model	Mixer	Params	MACs	img/s	Acc. (%)
ConvNeXt-T	C	29M	4.5G	3990	82.1
MambaOut-T	C	27M	4.5G	3031	82.7
Swin-T	A	29M	4.5G	2863	81.3
Twins-S	A	24M	2.9G	2669	81.7
EfficientFormer-L3	P+A	31M	3.9G	3246	82.4
FasterViT-0	C+A	31M	3.3G	5651	82.1

Table 1: Detailed comparison of image classification performance on ImageNet-1K. All models are evaluated with 224

\times

224 input resolution. In the token mixer type, C, P, A, and S denote Convolution, Pooling, Attention, and SSM, respectively.

Panel of Figure 4 — Figure 4: How much we can speedup the SSM calculation by changing $B_{1}$ . The four configurations of [batch size, dimension, state dimension, sequence length] are exact settings for ours-T stage 3, ours-T stage 4, ours-B stage 3, ours-B stage 4.

impl. opt.	BFold	$B_{1}$	conv	img/s
				6662
✓				6989
✓	✓	1	✓	7601
✓	✓	4	✓	7641
✓	✓	adaptive		7279
✓	✓	adaptive	✓	7685

Table 2: The computational speedup achieved by our method

swap	aux. token	discard timing	IN1K	ADE20K	img/s
			82.2	46.0	7645
	learnable	before attn	82.1	46.2	7613
✓	learnable	before attn	82.3	46.5	7585
✓	data-dependent	before attn	82.4	46.8	7602
✓	data-dependent	after 1st attn	82.5	47.2	7600
✓	data-dependent	after attn	82.4	46.6	7597

Table 3: The effectiveness of auxiliary token swapping and its ablation results.

macro-arch.	MambaVision-T				MambaVision-T w/o Attention
scan	Params	MACs	img/s	acc.	Params	MACs	img/s	acc.
uni-scan	31.8M	4.4G	6979	82.2	29.4M	4.2G	6238	80.2
series bi-scan	31.8M	4.4G	6911	82.3	29.4M	4.2G	6113	80.4
series bi-scan+gap token (Adventurer)	31.8M	4.5G	6856	82.3	29.4M	4.3G	6027	80.7
parallel bi-scan (cat)	31.8M	4.4G	6834	82.2	29.4M	4.2G	5987	80.8
parallel bi-scan (add)	31.9M	4.5G	6235	82.3	29.7M	4.3G	5138	81.1

Table 4: Comparison of effective scan designs. The “series bi-scan + gap token” follows Adventurer ( Wang et al., 2025b ) , where a global token obtained via global average pooling is used. In parallel bi-scan, ”cat” splits the input along channels, applies bi-scan, and concatenates the results, while ”add” duplicates the input, applies bi-scan, and sums the outputs. The ”Vim block” indicates exact Vim block is used instead of making MambaVision block bi-directional.

Figure 5: Throughput–accuracy trade-off on ADE20K. The x-axis denotes frames per second with batch size 1 setting (higher is better), and the y-axis shows mIoU (higher is better). SF-Mamba variants lie on the Pareto front.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	SF-Mamba-T	SF-Mamba-S	SF-Mamba-B
Optimizer	LAMB ( You et al., 2019 )	LAMB	LAMB
Base learning rate	5e-3	5e-3	5e-3
Learning rate schedule	Cosine	Cosine	Cosine
Warmup epochs	20	20	35
Warmup learning rate	1e-6	1e-6	1e-6
Minimum learning rate	5e-6	5e-6	5e-6

Appendix

Table 5: Training configurations for SF-Mamba variants on ImageNet-1K. All models are trained for 300 epochs following the MambaVision configuration ( Hatamizadeh and Kautz, 2025 ) .

Panel of Figure 6 — Figure 6: Computational cost of multi-directional scan. This includes the time required to reorder tokens for scanning from multiple directions, and the additional processing time incurred by setting up parallel paths.

Figure 7: Effective Receptive Field (ERF) comparison. This ERF corresponds to the layers up to the Stage 3 Mamba blocks. MambaVision uses a simple unidirectional scan, which prevents it from accessing future tokens (i.e., the lower part of the image) beyond what can be captured by convolution. In contrast, SF-Mamba leverages auxiliary token swapping, allowing it to account for both past and future tokens with similar strength. Since the auxiliary token swapping information propagation follows the mechanism of SSM, it can be effectively achieved with just two tokens.

Figure 8: Effective Receptive Field (ERF) comparison of the entire model. SF-Mamba achieves globally distributed ERFs with reduced computational complexity.

	Tiny-size				Small-size				Base-size
Backbone	Para	FLOPs	mIoU	fps	Para	FLOPs	mIoU	fps	Para	FLOPs	mIoU	fps
Swin	60M	945G	44.5	40.0	81M	1038G	47.6	25.7	121M	1188G	48.1	25.4
Focal	62M	998G	45.8	38.9	85M	1130G	48.0	24.0	126M	1354G	49.0	23.4
MambaVision	62M	1085G	46.0	45.0	81M	1166G	48.2	40.9	130M	1520G	49.1	37.3
SF-Mamba	62M	1085G	47.2	47.9	81M	1166G	48.5	45.4	130M	1520G	50.1	42.6
SF-Mamba $\clubsuit$	62M	950G	46.5	48.7	81M	1014G	48.1	47.3	130M	1180G	49.1	42.7

Appendix

Table 6: Semantic segmentation performance on ADE20K dataset using UperNet. We compare with Swin Transformer ( Liu et al., 2021 ) , Focal Transformer ( Yang et al., 2021 ) , and MambaVision ( Hatamizadeh and Kautz, 2025 ) . All models are trained at a resolution of 512

\times

512 while FLOPs are calculated with an input size of 2048

\times

512. Frames per second (FPS) are measured with a batch size of 1. SF-Mamba

\clubsuit

uses a windowed attention to save computational cost.

Backbone	Params	FLOPs	fps	AP b	AP ${}^{b}_{50}$	AP ${}^{b}_{75}$	AP m	AP ${}^{m}_{50}$	AP ${}^{m}_{75}$
Swin-T	86M	745G	26.3	50.4	69.2	54.7	43.7	66.6	47.3
ConvNeXt-T	86M	741G	32.1	50.4	69.1	54.8	43.7	66.5	47.3
MambaVision-T	89M	1118G	19.4	51.1	70.0	55.6	44.3	67.3	47.9
SF-Mamba-T	89M	741G	27.8	51.0	69.9	55.3	44.2	67.1	48.0
SF-Mamba $\clubsuit$ -T	89M	659G	28.3	50.9	69.9	55.0	44.1	66.9	47.7
Swin-S	107M	838G	18.7	51.9	70.7	56.3	45.0	68.2	48.8

Appendix

Table 7: Object detection and instance segmentation performance on MS COCO dataset using Cascade Mask R-CNN. All models are trained with 3

\times

schedule at 1280

\times

800 resolution. We compare with ConvNeXt ( Liu et al., 2022 ) , Swin Transformer ( Liu et al., 2021 ) , and MambaVision ( Hatamizadeh and Kautz, 2025 ) . SF-Mamba

\clubsuit

uses a windowed Attention (no window for Mamba blocks) to save computational cost.

Figure 9: Accuracy-speed trade-off on MS COCO using (a) Casecade Mask-RCNN ( Cai and Vasconcelos, 2019a ) and (b) Mask R-CNN ( He et al., 2017 ) . Casecade Mask-RCNN is trained with the 3x schedule while Mask-RCNN is trained with the 1x schedule. In MambaVision and in our model built on its macro-architecture, the tiny and small variants show a reversal in speed. This is because the tiny model has three Attention layers in stage 3 and two in stage 4, whereas the small model has two Attention layers in stage 3 and three in stage 4. When high-resolution images are used as input, Attention computation becomes the bottleneck. Consequently, even though the tiny model has fewer parameters, its throughput becomes lower due to the larger number of Attention layers in stage 3.

Backbone	AP box	fps
RegNetX-3.2GF	39.9	31.6
MambaVision-T	42.4	37.2
SF-Mamba-T	43.2	41.7
MambaVision-S	43.9	37.2
SF-Mamba-S	44.9	40.8
MambaVision-B	46.2	30.0

Appendix

Table 8: Detection on COCO dataset using Faster R-CNN. The models are trained with 1x schedule (12 epoch).

Backbone	AP ${}_{\text{box}}$	AP ${}_{\text{mask}}$	FPS
VMamba-T	47.3	42.7	29.9
MambaVision-T	43.1	40.0	35.5
SF-Mamba-T	43.8	40.3	40.9
VMamba-S	48.7	43.7	23.3
MambaVision-S	44.4	41.0	33.4
SF-Mamba-S	45.3	41.5	41.0

Appendix

Table 9: Detection and instance segmentation on COCO dataset using Mask R-CNN. The models are trained with 1x schedule (12 epoch).

		number of window				ADE20K		COCO
arch.	w/o pad	A3	A4	M3	M4	FLOPs	mIoU	FLOPs	mAP b	mAP m
w/o swap						1085G	46.0	1118G	50.9	44.1
w/ swap						1085G	47.2	1118G	51.2	44.5
w/ swap	✓					950G	45.8	741G	51.0	44.2
w/ swap		4				950G	46.2	741G	50.9	44.0
w/ swap	✓	4				942G	45.6	736G	50.9	44.0

Appendix

Table 10: The impact of excessive padding regions and the use of windowed Attention in terms of computational cost and accuracy. With the excessive padding setting, Stage 3 uses large padding sizes — 112

\times

112 for COCO and 64

\times

64 for ADE20K. In contrast, the w/o pad configuration minimizes padding to match the training image sizes, resulting in 84

\times

84 for COCO and 32

\times

32 for ADE20K. Regarding local windows: A3 refers to the windowed Attention in Stage 3 and M3 refers to the windowed Mamba in Stage 3. The configurations used for our models, SF-Mamba and SF-Mamba

\clubsuit

, are highlighted. For ADE20K, since large images are processed during testing, we retain the large padding.

size	scan	Params	MACs	img/s	acc.
S	parallel-bi scan (Vim)	26M	5.3G	1079	80.3
S	uni-scan	26M	4.9G	1639	79.3
S	uni-scan + swap (ours)	26M	5.0G	1614	80.1
S	uni-scan + swap (ours) + Bfold (ours)	26M	5.0G	2022	80.1
T	parallel-bi scan (Vim)	7M	1.5G	2094	76.3

Appendix

Table 11: Comparison on Vim-S macro-architecture. To match the parameter count, we increase the channel dimension from 384 to 400 in the uni-scan model. By applying our method to Vim-S, we can significantly improve its performance over Vim-T while maintaining inference speed comparable to Vim-T.

Model (batch size)	224	448	896	1792	3584
VMamba-T (bs=32)	1384	402	107	5	OOM
FasterViT-0 (bs=32)	1415	1400	418	99	OOM
MambaVision-T (bs=32)	3770	1578	324	50	5
SF-Mamba-T (bs=32)	3962	1777	397	61	6
SF-Mamba $\clubsuit$ -T (bs=32)	-	-	427	105	27
VMamba-T (bs=1)	62	62	62	24	5

Appendix

Table 12: Throughput (images/s) for different models and resolutions. OOM denotes out-of-memory with an A100 40GB GPU. We use a windowed Attention with a 32

\times

32 size for SF-Mamba

\clubsuit

, the same as the ADE20K setup. The ”-” for SF-Mamba

\clubsuit

means that the feature sizes of both stage 3 and 4 are less than 32

\times

32, so the same with SF-Mamba.

Arch.	1	32	128	256	512	1024
MambaVision-T	119	3770	6662	7025	7134	7271
ours-T	126	3962	7600	7801	8009	8190
MambaVision-B	96	2798	2974	3128	3176	3206
ours-B	98	3168	3534	3592	3641	3685

Appendix

Table 13: Throughput with various batch sizes

Arch.	Params	img/s	acc.
Attention only	34.2M	7803	82.3%
SSM only	29.4M	6238	80.2%
SSM only (w/ our Bfold and swap)	29.4M	7306	81.0%
Hybrid	31.8M	6979	82.2%
Hybrid (w/ our Bfold and swap)	31.8M	7600	82.5%

Appendix

Table 14: The contribution of Attention and Mamba.

Visual SSMs

Visual state-space models use recurrent sequence dynamics to process images and video while aiming for efficient, long-range context modeling.

See all 79 papers

MFil-Mamba: Multi-Filter Scanning for Spatial Redundancy-Aware Visual State Space Models
Mar 20, 2026Puskal Khadka, KC SantoshVisual SSMs Image Classification
USD Artifical Intelligence Research, Department of Computer Science, University of South Dakota, Vermillion, 57069, USA.
LiAuto-MindViT: A Hybrid Vision Backbone with Adaptive Bidirectional Mamba
Sep 21, 2026Lifu Mu, Shuai Chen, Wen Zheng +7Hybrid CNN-Transformer Architectures Visual SSMs
Li Auto Inc. · University of Science and Technology of China
Vision Non-Causal Trapezoidal Mamba: Eliminating Directional Scanning in Vision SSMs with Second-Order Dynamics
Jul 3, 2026Anvitha Ramachandran, Dhruv Parikh, Haoyang Fan +2Visual SSMs State Space Models
University of Southern California · DEVCOM Army Research Office

Mar 20, 2026J/K move · Enter open · S save

MFil-Mamba: Multi-Filter Scanning for Spatial Redundancy-Aware Visual State Space Models

Puskal Khadka, KC Santosh

USD Artifical Intelligence Research, Department of Computer Science, University of South Dakota, Vermillion, 57069, USA.

Visual SSMs Image Classification Semantic Segmentation

Open paper AskPDF arXiv

State Space Models (SSMs), especially recent Mamba architecture, have achieved remarkable success in sequence modeling tasks. However, extending SSMs to computer vision remains challenging due to the non-sequential structure of visual data and its complex 2D spatial dependencies. Although several early studies have explored adapting selective SSMs for vision applications, most approaches primarily depend on employing various traversal strategies over the same input. This introduces redundancy and distorts the intricate spatial relationships within images. To address these challenges, we propose MFil-Mamba, a novel visual state space architecture built on a multi-filter scanning backbone. Unlike fixed multi-directional traversal methods, our design enables each scan to capture unique and contextually relevant spatial information while minimizing redundancy. Furthermore, we incorporate an adaptive weighting mechanism to effectively fuse outputs from multiple scans in addition to architectural enhancements. MFil-Mamba achieves superior performance over existing state-of-the-art models across various benchmarks that include image classification, object detection, instance segmentation, and semantic segmentation. For example, our tiny variant attains 83.2% top-1 accuracy on ImageNet-1K, 47.3% box AP and 42.7% mask AP on MS COCO, and 48.5% mIoU on the ADE20K dataset. Code and models are available at https://github.com/puskal-khadka/MFil-Mamba.

Abstract

Visual SSMs

Explore similar work

MFil-Mamba: Multi-Filter Scanning for Spatial Redundancy-Aware Visual State Space Models

LiAuto-MindViT: A Hybrid Vision Backbone with Adaptive Bidirectional Mamba

Vision Non-Causal Trapezoidal Mamba: Eliminating Directional Scanning in Vision SSMs with Second-Order Dynamics