The realm of Mamba for vision has been advanced in recent years to strike for the alternatives of Vision Transformers (ViTs) that suffer from the quadratic complexity. While the recurrent scanning mechanism of Mamba offers computational efficiency, it inherently limits non-causal interactions between image patches. Prior works have attempted to address this limitation through various multi-scan strategies; however, these approaches suffer from inefficiencies due to suboptimal scan designs and frequent data rearrangement. Moreover, Mamba exhibits relatively slow computational speed under short token lengths, commonly used in visual tasks. In pursuit of a truly efficient vision encoder, we rethink the scan operation for vision and the computational efficiency of Mamba. To this end, we propose SF-Mamba, a novel visual Mamba with two key proposals: auxiliary patch swapping for encoding bidirectional information flow under an unidirectional scan and batch folding with periodic state reset for advanced GPU parallelism. Extensive experiments on image classification, object detection, and instance and semantic segmentation consistently demonstrate that our proposed SF-Mamba significantly outperforms state-of-the-art baselines while improving throughput across different model sizes. We will release the source code after publication.
Figures & tables
Figure 1: Top-1 accuracy and throughput on ImageNet-1K classification. SF-Mamba offers superior accuracy–throughput trade-offs compared to state-of-the-art architectures.
Figure 2: Future-to-Past Information Routing via Auxiliary Token Swapping. The left figure illustrates why the commonly used multi-directional scan in visual Mamba fails to achieve high speed, while the right figure presents our proposed solution. We prepend/append learnable auxiliary tokens to the patch sequence xheadaux and xtailaux . Within each MambaVision block, the causal selective scan aggregates sequence-wide context into the tail token ytailaux . A lightweight, parameter-free Swap operation then moves this global summary to the sequence head, yielding X~ for the next layer such that all patch states are conditioned on global context. It incurs negligible computational overhead while enabling effective global-context propagation across layers.
Figure 3: Batch folding with periodic state reset. ( Left ) An input tensor of shape [B,D,T] is reshaped into [B1,D,(B2⋅T)] , concatenating B2 short sequences into a longer one. This reshaping mixes hidden states across batches. ( Right ) To avoid information leakage, we reset the recurrence every T steps. Since ht←Atht−1+Btxt , setting At=0 at boundaries is equivalent to re-initializing the hidden state. In contrast, Bt (input projection) and Ct (output projection) operate locally and therefore remain unchanged.
Model
Mixer
Params
MACs
img/s
Acc. (%)
ConvNeXt-T
C
29M
4.5G
3990
82.1
MambaOut-T
C
27M
4.5G
3031
82.7
Swin-T
A
29M
4.5G
2863
81.3
Twins-S
A
24M
2.9G
2669
81.7
EfficientFormer-L3
P+A
31M
3.9G
3246
82.4
FasterViT-0
C+A
31M
3.3G
5651
82.1
Table 1: Detailed comparison of image classification performance on ImageNet-1K. All models are evaluated with 224 × 224 input resolution. In the token mixer type, C, P, A, and S denote Convolution, Pooling, Attention, and SSM, respectively.
Figure 4: How much we can speedup the SSM calculation by changing B1 . The four configurations of [batch size, dimension, state dimension, sequence length] are exact settings for ours-T stage 3, ours-T stage 4, ours-B stage 3, ours-B stage 4.
impl. opt.
BFold
B1
conv
img/s
6662
✓
6989
✓
✓
1
✓
7601
✓
✓
4
✓
7641
✓
✓
adaptive
7279
✓
✓
adaptive
✓
7685
Table 2: The computational speedup achieved by our method
swap
aux. token
discard timing
IN1K
ADE20K
img/s
82.2
46.0
7645
learnable
before attn
82.1
46.2
7613
✓
learnable
before attn
82.3
46.5
7585
✓
data-dependent
before attn
82.4
46.8
7602
✓
data-dependent
after 1st attn
82.5
47.2
7600
✓
data-dependent
after attn
82.4
46.6
7597
Table 3: The effectiveness of auxiliary token swapping and its ablation results.
macro-arch.
MambaVision-T
MambaVision-T w/o Attention
scan
Params
MACs
img/s
acc.
Params
MACs
img/s
acc.
uni-scan
31.8M
4.4G
6979
82.2
29.4M
4.2G
6238
80.2
series bi-scan
31.8M
4.4G
6911
82.3
29.4M
4.2G
6113
80.4
series bi-scan+gap token (Adventurer)
31.8M
4.5G
6856
82.3
29.4M
4.3G
6027
80.7
parallel bi-scan (cat)
31.8M
4.4G
6834
82.2
29.4M
4.2G
5987
80.8
parallel bi-scan (add)
31.9M
4.5G
6235
82.3
29.7M
4.3G
5138
81.1
Table 4: Comparison of effective scan designs. The “series bi-scan + gap token” follows Adventurer ( Wang et al., 2025b ) , where a global token obtained via global average pooling is used. In parallel bi-scan, ”cat” splits the input along channels, applies bi-scan, and concatenates the results, while ”add” duplicates the input, applies bi-scan, and sums the outputs. The ”Vim block” indicates exact Vim block is used instead of making MambaVision block bi-directional.
Figure 5: Throughput–accuracy trade-off on ADE20K. The x-axis denotes frames per second with batch size 1 setting (higher is better), and the y-axis shows mIoU (higher is better). SF-Mamba variants lie on the Pareto front.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
SF-Mamba-T
SF-Mamba-S
SF-Mamba-B
Optimizer
LAMB ( You et al., 2019 )
LAMB
LAMB
Base learning rate
5e-3
5e-3
5e-3
Learning rate schedule
Cosine
Cosine
Cosine
Warmup epochs
20
20
35
Warmup learning rate
1e-6
1e-6
1e-6
Minimum learning rate
5e-6
5e-6
5e-6
Appendix
Table 5: Training configurations for SF-Mamba variants on ImageNet-1K. All models are trained for 300 epochs following the MambaVision configuration ( Hatamizadeh and Kautz, 2025 ) .
Figure 6: Computational cost of multi-directional scan. This includes the time required to reorder tokens for scanning from multiple directions, and the additional processing time incurred by setting up parallel paths.
Figure 7: Effective Receptive Field (ERF) comparison. This ERF corresponds to the layers up to the Stage 3 Mamba blocks. MambaVision uses a simple unidirectional scan, which prevents it from accessing future tokens (i.e., the lower part of the image) beyond what can be captured by convolution. In contrast, SF-Mamba leverages auxiliary token swapping, allowing it to account for both past and future tokens with similar strength. Since the auxiliary token swapping information propagation follows the mechanism of SSM, it can be effectively achieved with just two tokens.
Figure 8: Effective Receptive Field (ERF) comparison of the entire model. SF-Mamba achieves globally distributed ERFs with reduced computational complexity.
Tiny-size
Small-size
Base-size
Backbone
Para
FLOPs
mIoU
fps
Para
FLOPs
mIoU
fps
Para
FLOPs
mIoU
fps
Swin
60M
945G
44.5
40.0
81M
1038G
47.6
25.7
121M
1188G
48.1
25.4
Focal
62M
998G
45.8
38.9
85M
1130G
48.0
24.0
126M
1354G
49.0
23.4
MambaVision
62M
1085G
46.0
45.0
81M
1166G
48.2
40.9
130M
1520G
49.1
37.3
SF-Mamba
62M
1085G
47.2
47.9
81M
1166G
48.5
45.4
130M
1520G
50.1
42.6
SF-Mamba ♣
62M
950G
46.5
48.7
81M
1014G
48.1
47.3
130M
1180G
49.1
42.7
Appendix
Table 6: Semantic segmentation performance on ADE20K dataset using UperNet. We compare with Swin Transformer ( Liu et al., 2021 ) , Focal Transformer ( Yang et al., 2021 ) , and MambaVision ( Hatamizadeh and Kautz, 2025 ) . All models are trained at a resolution of 512 × 512 while FLOPs are calculated with an input size of 2048 × 512. Frames per second (FPS) are measured with a batch size of 1. SF-Mamba ♣ uses a windowed attention to save computational cost.
Backbone
Params
FLOPs
fps
AP b
AP 50b
AP 75b
AP m
AP 50m
AP 75m
Swin-T
86M
745G
26.3
50.4
69.2
54.7
43.7
66.6
47.3
ConvNeXt-T
86M
741G
32.1
50.4
69.1
54.8
43.7
66.5
47.3
MambaVision-T
89M
1118G
19.4
51.1
70.0
55.6
44.3
67.3
47.9
SF-Mamba-T
89M
741G
27.8
51.0
69.9
55.3
44.2
67.1
48.0
SF-Mamba ♣ -T
89M
659G
28.3
50.9
69.9
55.0
44.1
66.9
47.7
Swin-S
107M
838G
18.7
51.9
70.7
56.3
45.0
68.2
48.8
Appendix
Table 7: Object detection and instance segmentation performance on MS COCO dataset using Cascade Mask R-CNN. All models are trained with 3 × schedule at 1280 × 800 resolution. We compare with ConvNeXt ( Liu et al., 2022 ) , Swin Transformer ( Liu et al., 2021 ) , and MambaVision ( Hatamizadeh and Kautz, 2025 ) . SF-Mamba ♣ uses a windowed Attention (no window for Mamba blocks) to save computational cost.
Figure 9: Accuracy-speed trade-off on MS COCO using (a) Casecade Mask-RCNN ( Cai and Vasconcelos, 2019a ) and (b) Mask R-CNN ( He et al., 2017 ) . Casecade Mask-RCNN is trained with the 3x schedule while Mask-RCNN is trained with the 1x schedule. In MambaVision and in our model built on its macro-architecture, the tiny and small variants show a reversal in speed. This is because the tiny model has three Attention layers in stage 3 and two in stage 4, whereas the small model has two Attention layers in stage 3 and three in stage 4. When high-resolution images are used as input, Attention computation becomes the bottleneck. Consequently, even though the tiny model has fewer parameters, its throughput becomes lower due to the larger number of Attention layers in stage 3.
Backbone
AP box
fps
RegNetX-3.2GF
39.9
31.6
MambaVision-T
42.4
37.2
SF-Mamba-T
43.2
41.7
MambaVision-S
43.9
37.2
SF-Mamba-S
44.9
40.8
MambaVision-B
46.2
30.0
Appendix
Table 8: Detection on COCO dataset using Faster R-CNN. The models are trained with 1x schedule (12 epoch).
Backbone
AP box
AP mask
FPS
VMamba-T
47.3
42.7
29.9
MambaVision-T
43.1
40.0
35.5
SF-Mamba-T
43.8
40.3
40.9
VMamba-S
48.7
43.7
23.3
MambaVision-S
44.4
41.0
33.4
SF-Mamba-S
45.3
41.5
41.0
Appendix
Table 9: Detection and instance segmentation on COCO dataset using Mask R-CNN. The models are trained with 1x schedule (12 epoch).
number of window
ADE20K
COCO
arch.
w/o pad
A3
A4
M3
M4
FLOPs
mIoU
FLOPs
mAP b
mAP m
w/o swap
1085G
46.0
1118G
50.9
44.1
w/ swap
1085G
47.2
1118G
51.2
44.5
w/ swap
✓
950G
45.8
741G
51.0
44.2
w/ swap
4
950G
46.2
741G
50.9
44.0
w/ swap
✓
4
942G
45.6
736G
50.9
44.0
Appendix
Table 10: The impact of excessive padding regions and the use of windowed Attention in terms of computational cost and accuracy. With the excessive padding setting, Stage 3 uses large padding sizes — 112 × 112 for COCO and 64 × 64 for ADE20K. In contrast, the w/o pad configuration minimizes padding to match the training image sizes, resulting in 84 × 84 for COCO and 32 × 32 for ADE20K. Regarding local windows: A3 refers to the windowed Attention in Stage 3 and M3 refers to the windowed Mamba in Stage 3. The configurations used for our models, SF-Mamba and SF-Mamba ♣ , are highlighted. For ADE20K, since large images are processed during testing, we retain the large padding.
size
scan
Params
MACs
img/s
acc.
S
parallel-bi scan (Vim)
26M
5.3G
1079
80.3
S
uni-scan
26M
4.9G
1639
79.3
S
uni-scan + swap (ours)
26M
5.0G
1614
80.1
S
uni-scan + swap (ours) + Bfold (ours)
26M
5.0G
2022
80.1
T
parallel-bi scan (Vim)
7M
1.5G
2094
76.3
Appendix
Table 11: Comparison on Vim-S macro-architecture. To match the parameter count, we increase the channel dimension from 384 to 400 in the uni-scan model. By applying our method to Vim-S, we can significantly improve its performance over Vim-T while maintaining inference speed comparable to Vim-T.
Model (batch size)
224
448
896
1792
3584
VMamba-T (bs=32)
1384
402
107
5
OOM
FasterViT-0 (bs=32)
1415
1400
418
99
OOM
MambaVision-T (bs=32)
3770
1578
324
50
5
SF-Mamba-T (bs=32)
3962
1777
397
61
6
SF-Mamba ♣ -T (bs=32)
-
-
427
105
27
VMamba-T (bs=1)
62
62
62
24
5
Appendix
Table 12: Throughput (images/s) for different models and resolutions. OOM denotes out-of-memory with an A100 40GB GPU. We use a windowed Attention with a 32 × 32 size for SF-Mamba ♣ , the same as the ADE20K setup. The ”-” for SF-Mamba ♣ means that the feature sizes of both stage 3 and 4 are less than 32 × 32, so the same with SF-Mamba.
Arch.
1
32
128
256
512
1024
MambaVision-T
119
3770
6662
7025
7134
7271
ours-T
126
3962
7600
7801
8009
8190
MambaVision-B
96
2798
2974
3128
3176
3206
ours-B
98
3168
3534
3592
3641
3685
Appendix
Table 13: Throughput with various batch sizes
Arch.
Params
img/s
acc.
Attention only
34.2M
7803
82.3%
SSM only
29.4M
6238
80.2%
SSM only (w/ our Bfold and swap)
29.4M
7306
81.0%
Hybrid
31.8M
6979
82.2%
Hybrid (w/ our Bfold and swap)
31.8M
7600
82.5%
Appendix
Table 14: The contribution of Attention and Mamba.
State Space Models (SSMs), especially recent Mamba architecture, have achieved remarkable success in sequence modeling tasks. However, extending SSMs to computer vision remains challenging due to the non-sequential structure of visual data and its complex 2D spatial dependencies. Although several early studies have explored adapting selective SSMs for vision applications, most approaches primarily depend on employing various traversal strategies over the same input. This introduces redundancy and distorts the intricate spatial relationships within images. To address these challenges, we propose MFil-Mamba, a novel visual state space architecture built on a multi-filter scanning backbone. Unlike fixed multi-directional traversal methods, our design enables each scan to capture unique and contextually relevant spatial information while minimizing redundancy. Furthermore, we incorporate an adaptive weighting mechanism to effectively fuse outputs from multiple scans in addition to architectural enhancements. MFil-Mamba achieves superior performance over existing state-of-the-art models across various benchmarks that include image classification, object detection, instance segmentation, and semantic segmentation. For example, our tiny variant attains 83.2% top-1 accuracy on ImageNet-1K, 47.3% box AP and 42.7% mask AP on MS COCO, and 48.5% mIoU on the ADE20K dataset. Code and models are available at https://github.com/puskal-khadka/MFil-Mamba.
Puskal Khadka, KC Santosh
USD Artifical Intelligence Research, Department of Computer Science, University of South Dakota, Vermillion, 57069, USA.
While Mamba-based models have shown strong potential for long sequence modeling, adapting them to vision is challenging due to the requirement of local neighborhood correlations and multi-directional spatial contexts for visual understanding. In this paper, we present LiAuto-MindViT, a novel hybrid vision backbone that synergizes the strengths of CNNs, Mamba, and Transformers. The core of our design is the Adaptive Bidirectional Mamba (ABM), which eliminates the directional bias of unidirectional SSMs through bidirectional selective scanning with learnable alpha blending, enabling content-adaptive directional fusion without the overhead of exhaustive multi-path routing. To further accelerate inference, we propose a deployment-friendly Reparameterized ConvSE (RepConvSE) module that leverages structural reparameterization to reduce latency and memory access overhead. Extensive experiments demonstrate that LiAuto-MindViT achieves state-of-the-art performance on image classification, object detection, and semantic segmentation while enabling efficient inference through reparameterization.
Lifu Mu, Shuai Chen, Wen Zheng +7
Li Auto Inc. · University of Science and Technology of China
State Space Models (SSMs) have emerged as an alternative to Vision Transformers, yet most vision SSMs inherit directional token scanning from causal sequence modeling. While effective for sequential data, directional scanning introduces spatial bias and orientation-sensitive representations. We present Vision Non-Causal Trapezoidal Mamba (VNCT), a second-order non-causal vision SSM that enables all image tokens to interact in a single pass, eliminating direSctional scanning and achieving low single-image inference latency. VNCT exhibits more orientation-robust representations, showing reduced performance degradation under image rotations and flips, while improving Boundary IoU by up to 3.7 points, leading to more accurate boundary preservation and object localization. Across ImageNet-1K classification, COCO object detection and instance segmentation, and ADE20K semantic segmentation, VNCT consistently outperforms both directional-scanning vision SSMs and first-order non-causal SSMs. These results show that directional scanning is unnecessary for high-performance vision SSMs and that second-order non-causal state-space modeling offers a simple, efficient, and robust alternative for visual recognition.
Anvitha Ramachandran, Dhruv Parikh, Haoyang Fan +2
University of Southern California · DEVCOM Army Research Office