cs.CVNov 5, 2026

Ultra-Lightweight 3D Medical Image Segmentation via Knowledge Distillation in Label-Scarce Domains

Authors: Rahim Ullah, Hasan Abbas, Syed Waqar Nabi

Organizations: School of Computing, University of Glasgow, Glasgow, United Kingdom · James Watt School of Engineering, University of Glasgow, Glasgow, United Kingdom

Abstract

Mamba is a neural network architecture that reduces computational complexity, and has been shown to be effective for 3D medical image segmentation. However, existing models still remain computationally expensive. We propose compressing Mamba by distillation while achieving competitive performance. Specifically, we demonstrate compression of a 139M-parameter SegMamba-V2 teacher into a lightweight TinyUNet3D student with about 0.96M parameters, achieving about 145×\times compression with a performance drop of just over one Dice point. We further investigate the utility of knowledge distillation in a range of data-labeling scenarios under identical experimental settings, and show that its advantage is most prominent when labeled data is scarce. To demonstrate the generalizability of our approach, experiments are conducted on three datasets: BraTS 2023, MSD Spleen and MSD Hippocampus. The code for TinyUNet3D is publicly available at: https://github.com/2802990u/tinyunet3d-dist-kd.

Figures & tables

Explore similar work

Nov 5, 2026

GMD-Mamba: Lightweight 3D Medical Image Segmentation Using a Grouped Multi-Directional Mamba

Mamba has the capability to capture global context in linear time for 3D medical image segmentation. However, existing methods employ Mamba with full-width directional scans at every resolution, keeping the model large (64-138M parameters and high FLOPs). We propose Grouped Multi-Directional (GMD) Mamba, a lightweight network based on the GMD scan that divides input channels CC into groups GG and flattens each group in different directions in one pass. This gives multi-directional context at O(C2/G)O(C^2/G) instead of O(K⋅C2)O(K \cdot C^2), where KK is the number of scan directions. Additionally, we employ Mamba only at the deepest encoder stages, resulting in 19.2M parameters. On BraTS2023, it achieves the best average Dice of 92.0% and the lowest HD95 of 2.7 at ∼\sim1/7 the parameters of SegMamba-V2. For model validation, we also ran experiments on the MSD Spleen dataset, achieving competitive performance. The implementation details and code are publicly available at https://github.com/2802990u/GMD-Mamba.
May 25, 2026

Detail Consistent Stage-Wise Distillation for Efficient 3D MRI Segmentation

Deploying high-performing 3D medical image segmenters (e.g., nnU-Net) is often limited by memory footprint and inference latency. Compression is therefore necessary, but compact 3D encoders tend to lose fine structural cues (small lesions and sharp boundaries) as downsampling repeats across multi-resolution stages. We propose Detail Consistent Distillation (DCD), a stage-wise distillation framework that preserves structural detail across scales by aligning teacher-student features in a wavelet-decomposed representation. At each encoder stage, DCD distills directional detail components in the wavelet domain while leaving the coarse approximation comparatively unconstrained, avoiding over-regularization of global semantics. DCD is used only during training and introduces no inference-time overhead. Experiments on the BraTS 2024 and ISLES 2022 benchmarks demonstrate that our approach achieves superior performance in MRI segmentation using 3D multi-modal data. Code and implementation details for DCD are publicly available at https://github.com/ClinicaAlpha/DCD-3D-MedSeg.
Jul 21, 2026

DAMamba-UNet3D: A Parameter-Efficient Mamba State Space U-Net with Dynamic Adaptive Scan for 3D Medical Image Segmentation

We propose parameter-efficient SSM-based U-Net architectures for 3D medical image segmentation. Convolutional U-Nets afford O(n) local mixing per layer but lack explicit global context; transformers provide global reasoning at O(n^2) cost in sequence length nn. State-space models (SSMs), such as Mamba, offer O(n)O(n) global propagation per block. Yet, existing medical SSM segmenters rely on fixed scan patterns and large parameter budgets. Dynamic Adaptive Scan (DAS), which learns data-dependent reordering before selective scan, has not been applied to medical imaging or extended to 3D volumes. We propose DAMamba-UNet3D, a hybrid encoder-decoder that integrates tri-plane 3D-DAS blocks at encoder stages E2-E4 while retaining convolutions elsewhere (~5.3M parameters). On BraTS 2020 five-fold cross-validation, DAMamba-UNet3D achieves mean Dice 0.815+/-0.013 (full-volume per-case evaluation) at ~13x lower parameter cost than SegMamba (0.824+-0.014, ~70M). At comparable scale, DAMamba-L (~70M), a wide DAS-native variant with encoder-only DAMamba and a convolutional bottleneck, reaches 0.829+-0.012, surpassing retrained SegMamba by 0.5pt. Component ablations show that encoder-only DAS placement is critical as bottleneck and decoder SSM blocks lower Dice. Together, the results suggest that learned tri-plane DAS in a hybrid U-Net is competitive with, and under our large-scale design may improve upon, SegMamba's fixed Tri-orientated Mamba (ToM) scanning on BraTS 2020. Code: https://github.com/marafathussain/DAMamba-UNet3D.