cs.CVApr 9, 2026

T-Gated Adapter: A Lightweight Temporal Adapter for Vision-Language Medical Segmentation

Authors: Pranjal Khadka

Organizations: Independent Researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Medical image segmentation traditionally relies on fully supervised 3D architectures that demand a large amount of dense, voxel-level annotations from clinical experts which is a prohibitively expensive process. Vision Language Models (VLMs) offer a powerful alternative by leveraging broad visual semantic representations learned from billions of images. However, when applied independently to 2D slices of a 3D scan, these models often produce noisy and anatomically implausible segmentations that violate the inherent continuity of anatomical structures. We propose a temporal adapter that addresses this by injecting adjacent-slice context directly into the model's visual token representations. The adapter comprises a temporal transformer attending across a fixed context window at the token level, a spatial context block refining within-slice representations, and an adaptive gate balancing temporal and single-slice features. Training on 30 labeled volumes from the FLARE22 dataset, our method achieves a mean Dice of 0.704 across 13 abdominal organs with a gain of +0.206 over the baseline VLM trained with no temporal context. Zero-shot evaluation on BTCV and AMOS22 datasets yields consistent improvements of +0.210 and +0.230, with the average cross-domain performance drop reducing from 38.0% to 24.9%. Furthermore, in a cross-modality evaluation on AMOS22 MRI with neither model receiving any MRI supervision, our method achieves a mean Dice of 0.366, outperforming a fully supervised 3D baseline (DynUNet, 0.224) trained exclusively on CT, suggesting that CLIP's visual semantic representations generalize more gracefully across imaging modalities than convolutional features.

Figures & tables

Figure 1: Temporal inconsistency of slice-wise VLM inference on the pancreas. Green slice labels indicate the organ is present in ground truth. The baseline (middle row) produces spurious detections in anatomically implausible slices. Our method (bottom row) remains consistent, suppressing predictions where the pancreas is absent.
Figure 2: Overview of the proposed temporal adapter. Novel components include temporal transformer, spatial context block, and adaptive gate which are highlighted in blue and green within the Temporal Adapter module.
OrganWeight
R. Adrenal, L. Adrenal×8.0\times 8.0
Duodenum, Esophagus×8.0\times 8.0
Pancreas, Stomach×2.0\times 2.0
Gallbladder×2.0\times 2.0
Liver, Spleen, Kidneys×1.0\times 1.0
Aorta, IVC×1.0\times 1.0
Table 1: Class-imbalanced sampling weights
MethodFLARE22BTCVAMOS22 CT
CLIPSeg Baseline0.4970.3340.283
CLIPSeg + Temporal (Ours)0.7040.5440.513
Table 2: Mean Dice across 13 abdominal organs on CT benchmarks. BTCV and AMOS22 CT are zero-shot with no model adaptation. Average drop computed relative to FLARE22. Both models trained on 30 labeled volumes.
Figure 3: Qualitative segmentation comparison on the best-performing slice per organ. Dice scores are shown above each prediction column.
OrganBaseline+ TemporalΔ\Delta Dice
Liver0.9110.960+0.049+0.049
Spleen0.6910.919+0.228+0.228
Pancreas0.2430.647+0.404+0.404
Stomach0.5810.849+0.268+0.268
Aorta0.7060.877+0.171+0.171
Gallbladder0.4420.715+0.273+0.273
Table 3: Per-organ Dice on FLARE22 test set (10 volumes). Mean is computed as volume-weighted average across all organs and volumes.
ConditionPrompt ExampleMean Dice
Correct prompt"liver" →\to liver GT0.7040.704
Blank prompt"" →\to liver GT0.0050.005 ( −99.3%-99.3\% )
Wrong prompt"aorta" →\to liver GT0.0110.011 ( −98.4%-98.4\% )
Table 4: Text prompt sensitivity on FLARE22 test set. Corrupting the text prompt causes near-total collapse, confirming that the model is conditioned on the language query.
Figure 4: Zero-shot cross-modality generalization on AMOS22 MRI (right kidney) with neither model receiving any MRI supervision.
MethodAMOS22 MRI
DynUNet0.224
CLIPSeg + Temporal Adapter0.366
Table 5: Zero-shot cross-modality evaluation on AMOS22 MRI (10 volumes) with neither model receiving any MRI supervision
More in this topic

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 788 papers

Explore similar work

CardsList
Aug 13, 2026

MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation

Medical image segmentation is still largely treated as a vision-only problem, although clinical interpretation often relies on textual knowledge of anatomy, location, appearance, and surrounding context. Existing text-guided segmentation methods within the Vision-Language Model (VLM) paradigm often use language only as a late conditioning signal, limiting its influence on visual representation learning. We introduce MedPlex (Medical Plexus of Vision and Language), an end-to-end VLM framework that makes text guidance a continuous, clinically grounded component of segmentation learning. Through Bi-Fusion (Bidirectional Fusion), visual and textual representations evolve jointly across the encoding hierarchy. MedPlex further introduces class-level and region-level concept alignment to organize the shared representation at complementary granularities. Class-level alignment anchors each anatomical target to an aggregated clinical concept profile, while region-level alignment preserves individual concepts, such as shape, location, appearance, and texture, through class-specific visual evidence. In this way, language provides structured supervision throughout the encoder rather than serving only as a late-stage cue. MedPlex achieves state-of-the-art performance across CT and MR benchmarks for multi-organ, cardiac substructure, and tumor segmentation, including settings with real free-text clinical supervision. Code: https://github.com/rafiibnsultan/MedPlex.

Rafi Ibn Sultan, Hui Zhu, Chengyin Li +1

Department of Computer Science Wayne State University · Department of Radiation Oncology Henry Ford Health · Institute for AI and Data Science Wayne State University

Medical Image SegmentationText-Guided Image SegmentationMedical VLMs
AskPDF
Feb 23, 2026

MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation

Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision-language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages more nuanced semantic learning across diverse textual prompts, MedCLIPSeg effectively improves data efficiency and domain generalizability. Extensive experiments across 16 datasets spanning five imaging modalities and six organs demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight local reliability of segmentation results. This work demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.

Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari +3

Concordia University, Montreal, Canada

Medical Image SegmentationText-Guided Image SegmentationMedical VLMsVLM RobustnessSegmentation Uncertainty Estimation
AskPDF
Jul 20, 2026

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation

Test-time adaptation (TTA) aims to mitigate distribution shifts by adapting models with unlabeled target data at inference time. While TTA with vision-language models (VLMs) has shown promising results in classification, extending it to medical image segmentation remains challenging. In this setting, the adaptation gains from optimizing on VLM-generated predictions are often outweighed by the degradation to the VLM's strong pretrained features caused by noisy, update-driven learning, resulting in limited and unstable improvements. We therefore propose Memory-Supported Synergistic Adaptation (MSSA), a novel training-free TTA framework for medical image segmentation. Without updating model parameters, MSSA dynamically selects reliable image-text predictions to construct an online memory, uses them as text-guided semantic priors, and couples them with cross-image structural alignment for robust adaptation. Specifically, MSSA consists of (i) a noise-aware memory construction module that filters and stabilizes cross-modal predictions, and (ii) a relevance-driven prototype alignment module that aligns the target sample with structurally consistent memory samples and their reliable predictions to improve adaptation. Extensive experiments on multiple medical segmentation benchmarks demonstrate that MSSA consistently improves VLM-based segmentation models and outperforms existing fine-tuning-based TTA methods by a clear margin, with gains of up to 12.2% DSC and 11.7% mIoU. Project page: https://lingrayy.github.io/MSSA/ .

Lingrui Li, Nan Pu, Dong Zhao +4

School of Computer Science, University of Nottingham, UK · School of Computer Science and Information Engineering, Hefei University of Technology, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore

Memory-Augmented VLMsVLM AdaptationText-Guided Image SegmentationMedical Image Segmentation
AskPDF