Parameter-Efficient Fine-Tuning of Foundation Models for Liver Tumor Segmentation in CT
Authors: Ramtin Mojtahedi, Mohammad Hamghalam, Jacob J. Peoples, Richard K. G. Do, Amber L. Simpson
Organizations: School of Computing, Queen’s University, Kingston, ON, Canada · Toronto General Research Institute, University Health Network, Toronto, ON, Canada · Department of Electrical Engineering, Qa.C., Islamic Azad University, Qazvin, Iran · Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA · Department of Radiology and Diagnostic Imaging, University of Alberta, Edmonton, AB, Canada · Alberta Machine Intelligence Institute, Edmonton, AB, Canada
We evaluated parameter-efficient fine-tuning (PEFT) of the Segment Anything Model (SAM) for liver tumor segmentation in abdominal CT of colorectal liver metastases. We compared Low-Rank Adaptation (LoRA), 4-bit Quantized LoRA (QLoRA), a convolutional adapter (Conv-Adapter), and our Directional Spectral Top-K adapter (DiSCo), training only adapters while freezing the SAM backbone. DiSCo derives spectral bases from singular value decomposition of row-normalized weights and learns rank-gated spectral coefficients, per-output magnitude offsets, and a spectral gain, with optional Top-K rank selection at inference and 0.14 M trainable parameters. We benchmarked five prompting regimes: no prompt, single-point, multi-point, and bounding boxes at intersection over union 0.50 and 0.75. Conv-Adapter and LoRA achieved the highest accuracy (overall Dice 0.793 and 0.792; single-point Dice 0.795 and 0.792; 95th-percentile Hausdorff distance (HD95) 32 mm). QLoRA was close (overall Dice 0.766; single-point Dice 0.768; HD95 36.41 mm), with 0.91 M trainable parameters, 120 ms latency, and 4.9 GB peak memory. DiSCo achieved the highest Dice per million trainable parameters (4.66), with overall Dice 0.653, single-point Dice 0.698, and HD95 49.53 mm. These results show an accuracy-efficiency trade-off and support PEFT for liver tumor segmentation with reduced adaptation costs when compute and labeled data are limited. Code: https://github.com/Ramtin-Mojtahedi/PEFT-SAM-Liver-CT
Figures & tables
Figure 1: PEFT on SAM for liver tumor segmentation in CRLM CT: we froze SAM encoders, trained only Conv-Adapter, LoRA, QLoRA, and SVD-based DiSCo adapters in the encoder or decoder while the mask decoder fused point or box prompts with image embeddings, and we evaluated multiple prompts using Dice, HD95, and compute metrics.
Figure 2: Adapter accuracy–efficiency summary. (a) Mean Dice: Conv-Adapter/LoRA highest; QLoRA lower; DiSCo lowest. (b) Accuracy vs. latency: QLoRA fastest; DiSCo highlighted. (c) Accuracy vs. trainable parameters (log): Conv-Adapter/LoRA at higher capacity; QLoRA cuts parameters with moderate Dice drop; DiSCo in the extreme low-parameter regime. (d) Dice per million trainable parameters: DiSCo shows the best capacity efficiency.
Adapter
Dice ↑
Single-pt ↑
HD95 (mm) ↓
Lat. (ms) ↓
Mem (GB) ↓
Train. (M) ↓
FLOPs (TFLOPs/img) ↓
Dice/M ↑
DiSCo
0.6530
0.6983
49.53
143.93
5.47
0.14
0.2807
4.657
Conv-Adapter
0.7932
0.7953
32.01
131.95
5.34
8.72
0.1860
0.091
LoRA
0.7919
0.7918
31.74
145.38
6.09
11.31
0.9000
0.070
QLoRA
0.7655
0.7680
36.41
119.97
4.94
0.91
0.0020
0.842
Table 1: Combined performance and resource summary on adapters. Dice and HD95 are patient-level means averaged across all prompt regimes; Single-pt reports the single-point regime. Resource metrics include inference latency, peak memory, trainable parameters, and FLOPs per image (TFLOPs/img); Dice/M reports Dice per million trainable parameters.
Accurate segmentation of colorectal liver metastases (CRLM) in contrast-enhanced computed tomography (CT) is important for response assessment, surgical planning, and follow-up. We propose two parameter-efficient spectral adapters for the Segment Anything Model (SAM): the Directional Spectral Adapter (DiSECT) and Spectral Instance-Guided Adapter (SiGA). DiSECT uses singular value decomposition of frozen weights to constrain residual updates to leading spectral directions, while SiGA adds global and input-conditioned gating through a multilayer perceptron. We evaluate these methods on 446 contrast-enhanced CT volumes (355 training, 91 testing) and compare them with LoRA, QLoRA, convolutional adapters (CAD), and a 3D nnU-Net baseline. Experiments consider single-point, three-point, bounding-box, and no-prompt regimes. SiGA achieves the best single-point performance with a Dice score of 0.77, IoU of 0.69, and HD95 of 35.39 mm. Under no-prompt inference, SiGA reaches 0.76 Dice, 0.68 IoU, and 46.76 mm HD95, comparable to the nnU-Net baseline (0.758 Dice). DiSECT uses only 0.14 million trainable parameters. These results show that spectral adapters can efficiently adapt SAM for CRLM segmentation while retaining strong accuracy with limited trainable parameters.
Ramtin Mojtahedi, Mohammad Hamghalam, Jacob J. Peoples +6
School of Computing, Queen’s University, Kingston, ON, Canada · Toronto General Hospital Research Institute, University Health Network, 200 Elizabeth Street, Toronto, ON M5G 2C4, Canada · Department of Electrical Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran +6
Accurate brain tumor segmentation from magnetic resonance imaging (MRI) is essential for diagnosis and treatment planning; however, manual delineation is labor-intensive and time-consuming. Existing approaches are either task-specific with limited generalization or rely on user-provided prompts during inference. We present BaT-SAM, a parameter-efficient adaptation framework that enables fully automatic brain tumor segmentation with the Segment Anything Model (SAM). BaT-SAM injects local inductive bias into SAM's frozen Vision Transformer (ViT) encoder via Conv-LoRA, where image features are processed by a sparsely gated mixture of multi-scale convolutional experts. This design compensates for the limited translation-equivariant inductive bias of vanilla ViTs while fine-tuning fewer than 5M parameters (less than 1% of the full model). To remove the need for user prompts, we further introduce a lightweight self-prompting box regressor that is jointly optimized with the adapter using scheduled sampling and predicts tumor locations directly from image embeddings. On the BRISC2025 benchmark, BaT-SAM achieves a weighted mean IoU of 0.832 while outperforming state-of-the-art benchmarks. It further demonstrates strong cross-dataset generalization, achieving Dice scores of 0.702 on Figshare and 0.784 on the LGG MRI Segmentation dataset.
Sumiaya Jannat, Md Mahedi Hasan, Md Shamimul Islam +1
Mymensingh Medical College Mymensingh, Bangladesh · Department of LCSEE West Virginia University Morgantown, WV, USA · Department of CS Florida Polytechnic University Lakeland, Florida, USA +1
Efficient surgical segmentation empowers clinical diagnosis, intraoperative monitoring, and downstream robotic pipelines for reconstruction and simulation. Although prompt-driven foundation models like Segment Anything Model 3 (SAM3) achieve strong segmentation performance on natural images, surgical data exhibits domain gaps against its pre-training data, resulting in degraded segmentation accuracy. Furthermore, existing medical SAM methods require full-parameter fine-tuning, incurring heavy computational consumption and low efficiency. To address these limitations, this work proposes a parameter-efficient Low-Rank Adaptation (LoRA) adaptation of SAM3 for surgical concept segmentation. We inject low-rank adapters into the prompt encoder, detector and tracker while fully freezing the vision backbone, which only optimizes 0.98% of the total model parameters and supports training on a single consumer GPU. Comprehensive experiments demonstrate that our method consistently outperforms zero-shot SAM3 and other mainstream baselines, and the generated segmentation results can be directly deployed to support downstream robotic surgical scene reconstruction and physical simulation pipelines.
Changjing Liu, Yiming Huang, Beilei Cui +5
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · XGEN Labs