BaT-SAM: Introducing Inductive Bias into SAM via a Mixture of Convolutional Experts for Brain Tumor Segmentation

Sumiaya Jannat; Md Mahedi Hasan; Md Shamimul Islam; Md Nur Hossain

cs.CVOct 11, 2026

BaT-SAM: Introducing Inductive Bias into SAM via a Mixture of Convolutional Experts for Brain Tumor Segmentation

Authors: Sumiaya Jannat, Md Mahedi Hasan, Md Shamimul Islam, Md Nur Hossain

Organizations: Mymensingh Medical College Mymensingh, Bangladesh · Department of LCSEE West Virginia University Morgantown, WV, USA · Department of CS Florida Polytechnic University Lakeland, Florida, USA · Department of CSE Manarat International University Dhaka, Bangladesh

Abstract

Accurate brain tumor segmentation from magnetic resonance imaging (MRI) is essential for diagnosis and treatment planning; however, manual delineation is labor-intensive and time-consuming. Existing approaches are either task-specific with limited generalization or rely on user-provided prompts during inference. We present BaT-SAM, a parameter-efficient adaptation framework that enables fully automatic brain tumor segmentation with the Segment Anything Model (SAM). BaT-SAM injects local inductive bias into SAM's frozen Vision Transformer (ViT) encoder via Conv-LoRA, where image features are processed by a sparsely gated mixture of multi-scale convolutional experts. This design compensates for the limited translation-equivariant inductive bias of vanilla ViTs while fine-tuning fewer than 5M parameters (less than 1% of the full model). To remove the need for user prompts, we further introduce a lightweight self-prompting box regressor that is jointly optimized with the adapter using scheduled sampling and predicts tumor locations directly from image embeddings. On the BRISC2025 benchmark, BaT-SAM achieves a weighted mean IoU of 0.832 while outperforming state-of-the-art benchmarks. It further demonstrates strong cross-dataset generalization, achieving Dice scores of 0.702 on Figshare and 0.784 on the LGG MRI Segmentation dataset.

Figures & tables

Fig. 1: Overview of BaT-SAM. (a) Conv-LoRA adapters are inserted into the frozen SAM ViT-H encoder. (b) Each Conv-LoRA adapter routes the rank- $r$ bottleneck through a sparsely gated mixture of multi-scale convolutional experts, injecting local inductive bias absent in standard LoRA. (c) A self-prompting box regressor, consisting of a convolutional stem, global average pooling, and a lightweight MLP, predicts the bounding-box prompt for the prompt encoder.

Class	Train / Test	Area (%)	$x$ ( $\mu\pm\sigma$ )	$y$ ( $\mu\pm\sigma$ )
Glioma	1147 / 254	$1.87\pm 1.48$	$0.49\pm 0.11$	$0.43\pm 0.11$
Meningioma	1329 / 306	$2.25\pm 1.83$	$0.50\pm 0.13$	$0.44\pm 0.15$
Pituitary	1457 / 300	$1.04\pm 0.73$	$0.50\pm 0.05$	$0.51\pm 0.08$
Total	3933 / 860	–	–	–

TABLE I: Statistics of the BRISC2025 segmentation split by tumor class. Location and shape statistics are computed over all annotated masks; location normalized centroid (

x,y\in[0,1]

Generalization	Evaluation	Method	Base	Type	Dice	IoU
in-domain	Fine-tuning	Swin-HAFUNet [ 11 ]	UNet variant	BRISC2025 $\to$ BRISC2025	0.8624	0.8240
in-domain	Fine-tuning	BaT-SAM (ours)	SAM	BRISC2025 $\to$ BRISC2025	0.8865	0.8325
in-domain	Fine-tuning	Attention U-Net [ 29 ]	UNet variant	LGG $\to$ LGG	0.9016	0.8078
in-domain	Fine-tuning	Res-UNet [ 14 ]	UNet variant	LGG $\to$ LGG	0.8530	0.7590
in-domain	Fine-tuning	BaT-SAM (ours)	SAM	LGG $\to$ LGG	0.8561	0. 8090
in-domain	Fine-tuning	Residual-UNet [ 3 ]	UNet variant	Figshare $\to$ Figshare	0.7369	0.6151

TABLE II: Comparison of BaT-SAM on BRISC2025, LGG MRI, and Figshare brain tumor dataset for in-domain and cross-domain evaluation.

Fig. 2: Qualitative segmentation results of BaT-SAM on the BRISC2025 test set. One representative example is shown for each tumor type across the axial, coronal, and sagittal planes. Each triplet consists of the input MRI slice, the ground-truth mask, and the corresponding BaT-SAM prediction.

Model	Glioma	Meningioma	Pituitary	mIoU
UNet [ 25 ]	0.697	0.771	0.793	0.757
UNet++ [ 32 ]	0.717	0.742	0.797	0.753
MANet [ 10 ]	0.724	0.775	0.780	0.762
LinkNet [ 6 ]	0.717	0.748	0.790	0.753
DeepLabV3+ [ 7 ]	0.720	0.775	0.787	0.763
PAN [ 18 ]	0.720	0.745	0.807	0.759

TABLE III: IoU for brain tumor segmentation models on BRISC2025, by tumor type. mIoU is the sample-count-weighted average across tumor types.

Explore similar work

LoGSAM: Parameter-Efficient Cross-Modal Grounding for MRI Segmentation
Mar 18, 2026Mohammad Robaitul Islam Bhuiyan, Sheethal Bhat, Melika Qahqaie +4Image Segmentation Medical Imaging Foundation Models
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Siemens Healthineers, Erlangen, Germany
Parameter-Efficient Fine-Tuning of Foundation Models for Liver Tumor Segmentation in CT
Sep 12, 2026Ramtin Mojtahedi, Mohammad Hamghalam, Jacob J. Peoples +2Medical Image Segmentation Parameter-Efficient Fine-Tuning
School of Computing, Queen’s University, Kingston, ON, Canada · Toronto General Research Institute, University Health Network, Toronto, ON, Canada · Department of Electrical Engineering, Qa.C., Islamic Azad University, Qazvin, Iran +3
Confidence-Calibrating Regularization for Robust Brain MRI Segmentation Under Domain Shift
Sep 27, 2025Behraj Khan, Tahir Qasim Syed, Syed Ahmad Chan BukhariImage Segmentation Promptable Image Segmentation
St. John’s University New York, USA · Institute of Business Administration Karachi Pakistan

Mar 18, 2026J/K move · Enter open · S save

LoGSAM: Parameter-Efficient Cross-Modal Grounding for MRI Segmentation

Mohammad Robaitul Islam Bhuiyan, Sheethal Bhat, Melika Qahqaie, Andreas Maier+3

Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Siemens Healthineers, Erlangen, Germany

Image Segmentation Medical Imaging Foundation Models Text-Guided Image Segmentation Medical Image Segmentation Medical Image Grounding Brain Tumor Segmentation

Open paper AskPDF arXiv

Precise localization and delineation of brain tumors using magnetic resonance imaging (MRI) are essential for planning therapy and guiding surgical decisions. To address this, we propose LoGSAM, a parameter-efficient, detection-driven framework that transforms radiologist dictation into text prompts for foundation-model-based localization and segmentation. Radiologist speech is first transcribed and translated using a pretrained Whisper ASR model, followed by negation-aware clinical NLP to extract tumor-specific textual prompts. These prompts guide text-conditioned tumor localization via a LoRA-adapted vision-language detection model, Grounding DINO (GDINO). The predicted bounding boxes are used as prompts for MedSAM to generate pixel-level tumor masks without any additional fine-tuning. On BRISC 2025, LoGSAM attains a Dice score of 80.32%, reaching 98.6% of a fully fine-tuned GDINO + MedSAM baseline while training fewer than 5% of its parameters, indicating a favorable accuracy/parameter trade-off. In addition, we evaluate the full pipeline using German dictations from a board-certified radiologist on unseen MRI scans, achieving 91.7% case-level class-extraction accuracy. These results highlight the feasibility of constructing a modular speech-to-segmentation pipeline from pretrained foundation models with minimal parameter updates.

Abstract

Explore similar work

LoGSAM: Parameter-Efficient Cross-Modal Grounding for MRI Segmentation

Parameter-Efficient Fine-Tuning of Foundation Models for Liver Tumor Segmentation in CT

Confidence-Calibrating Regularization for Robust Brain MRI Segmentation Under Domain Shift