Prompt-Conditioned Channel Attention for Hierarchical Feature Modulation toward Anatomy-Agnostic Segmentation

Mosharof Hossain; Md Rabiul Islam; Limon Halder; Erchin Serpedin; Md Kamrul Hasan

cs.CVAug 20, 2026

Prompt-Conditioned Channel Attention for Hierarchical Feature Modulation toward Anatomy-Agnostic Segmentation

Authors: Mosharof Hossain, Md Rabiul Islam, Limon Halder, Erchin Serpedin, Md Kamrul Hasan

Organizations: Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology (KUET), Khulna-9203, Bangladesh · Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA · Department of Bioengineering, Imperial College London, London SW7 2AZ, UK

Abstract

Anatomically plausible segmentation remains challenging because of low contrast, ambiguous boundaries, and modality-specific artifacts. Interactive segmentation has emerged as a promising strategy to guide feature extraction and improve localization, particularly in structurally ambiguous regions. However, existing methods integrate prompts through late-stage fusion and lack explicit mechanisms for prompt-driven channel-wise modulation across hierarchical feature representations, limiting their ability to capture deeper contextual and modality-specific variations. To address these limitations, we introduce Prompt-Conditioned Channel Attention (PCCA), a novel modulation mechanism that enables deep, hierarchical integration of semantic prompts within encoder-decoder networks. PCCA extracts compact channel descriptors via pooling, projects them into a shared space, and fuses them through a gated excitation mechanism to compute prompt-aware channel attention weights. These weights adaptively recalibrate feature responses across multiple network stages, enabling prompt-conditioned, semantically enriched hierarchical representations. Building on this, we propose PROMISE-Net, instantiated in two network variants: a convolutional model (PROMISE-CNN) and a transformer-based model (PROMISE-Txformer). Across the ISIC-Lesion, Kvasir-Polyp, CAMUS-Cardiac, and Kvasir-Instrument benchmarks, integrating PCCA into PROMISE-CNN yielded relative IoU gains of 10.4%, 8.7%, 0.8%, and 3.4%, respectively, over the baseline U-Net, while PROMISE-Txformer achieved corresponding gains of 7.6%, 23.0%, 2.1%, and 1.1%, respectively, over the baseline UNETR. These results show consistent improvements across architectures, imaging modalities, and anatomical targets, establishing PCCA and PROMISE-Net as a scalable, generalizable framework for prompt-aware hierarchical feature modulation in medical image segmentation.

Figures & tables

Figure 1: From static feature learning to hierarchical prompt-conditioned feature control. Compared with U-Net and SAM, which rely on static representations or shallow prompt fusion, PROMISE-Net integrates Prompt-Conditioned Channel Attention (PCCA) throughout the network, enabling dynamic channel-wise gating and spatial–semantic modulation for improved segmentation performance.

Figure 2: Overview of PROMISE-Net architecture. The top panel presents the proposed PCCA module, while the bottom panel illustrates the segmentation backbone with hierarchical PCCA integration. Embedding PCCA within convolutional blocks yields PROMISE-C NN , while analogous integration in transformer blocks results in PROMISE-T xformer .

Figure 3: Qualitative comparison of ISIC 2017 skin lesion segmentation. PROMISE-Net with hierarchical PCCA yields more complete and spatially coherent lesion delineations compared to UNETR and other baselines, demonstrating improved boundary precision and region completeness.

Methods	Params ( $\downarrow$ )	FLOPs ( $\downarrow$ )	ISIC 2017		CAMUS
			DSC ( $\%$ ) ( $\uparrow$ )	FNR ( $\%$ ) ( $\downarrow$ )	DSC ( $\%$ ) ( $\uparrow$ )	FNR ( $\%$ ) ( $\downarrow$ )
(1) UNETR Hatamizadeh et al. [2022]	116,048,130	52,792,357,120	$82.5\pm 18.0$	$19.0\pm 21.5$	$88.6\pm 5.4$	$11.6\pm 7.4$
(2) UNETR + PCCA in bottleneck	127,922,994	58,231,075,072	$88.3\pm 9.6$	$12.7\pm 13.1$	$89.8\pm 4.7$	$10.5\pm 6.2$
(3) UNETR + PCCA in bottleneck and decoder	128,357,120	60,044,940,800	$\boldsymbol{89.0\pm 8.2}$	$10.6\pm 10.9$	$\boldsymbol{90.0\pm 5.1}$	$\boldsymbol{9.2\pm 5.8}$
(4) UNETR + OPF in bottleneck and decoder	127,055,010	60,042,342,656	$87.4\pm 9.4$	$\boldsymbol{10.2\pm 11.4}$	$89.6\pm 5.0$	$9.8\pm 6.1$
(1) vs. (2) ( $\mathbf{p<0.05}$ ?)			✓	✓	✓	✓

Table 1: Ablation study of PCCA integration on the ISIC-2017 and CAMUS datasets. Progressive insertion of PCCA into deeper UNETR stages improves DSC and reduces FNR with modest computational overhead, demonstrating that hierarchical prompt-guided modulation enhances feature discrimination and robustness. Bold blue values denote the best performance.

Panel of Figure 4 — Figure 4: Ablation visualization in two-column layout: PROMISE-Net with hierarchical PCCA yields more complete delineations and more spatially focused feature representations than baseline UNETR and OPF fusion.

Reduction Ratio ( $r$ )	HD95 ( $\downarrow$ )	FNR ( $\%$ ) ( $\downarrow$ )	FLOPs ( $\downarrow$ )	Params ( $\downarrow$ )
$r=2$	12.9	10.5	106.33347	53,070,338
$r=4$	12.9	11.2	106.33242	52,545,794
$r=8$	12.9	10.9	106.33190	52,283,522
$r=16$	$\boldsymbol{12.8}$	$\boldsymbol{10.3}$	106.33164	52,152,386
$r=32$	12.7	10.6	106.33150	52,086,818

Table 2: Effect of the reduction ratio

r

on PCCA efficiency and performance on ISIC-2017. A moderate setting (

r{=}16

) offers the best accuracy–efficiency trade-off, achieving low HD95 and FNR with minimal overhead. Bold blue indicates the best result.

Methods	DSC ( $\%$ ) ( $\uparrow$ )	IoU ( $\%$ ) ( $\uparrow$ )	HD95 (Pix) ( $\downarrow$ )	FNR ( $\%$ ) ( $\downarrow$ )
Skin Lesion Dataset (ISIC-2017) Codella et al. [2017]
(1) U-Net Ronneberger et al. [2015]	$82.2\pm 18.8$	$73.3\pm 21.8$	$23.6\pm 26.4$	$19.4\pm 21.9$
(2) PROMISE-C NN (proposed)	$\boldsymbol{90.7\pm 7.5}$	$\boldsymbol{83.7\pm 10.0}$	$\boldsymbol{11.7\pm 10.6}$	$\boldsymbol{7.3\pm 9.0}$
(1) vs. (2) ( $\mathbf{p<0.05}$ ?)	✓	✓	✓	✓
(3) UNETR Hatamizadeh et al. [2022]	$82.5\pm 18.0$	$73.4\pm 21.4$	$22.3\pm 25.2$	$19.0\pm 21.5$
(4) PROMISE-T xformer (proposed)	$89.0\pm 8.2$	$81.0\pm 11.4$	$15.7\pm 18.2$	$10.6\pm 10.9$

Table 3: Performance comparison across four segmentation datasets: ISIC-2017 (skin lesion), Kvasir-Polyp, Kvasir-Instrument (endoscopic tool), and CAMUS (cardiac echocardiography). PROMISE-Net consistently outperforms baseline architectures across diverse anatomical structures and imaging modalities, demonstrating strong cross-domain and cross-architectural generalizability. Bold blue values indicate the best performance for each metric.

Panel of Figure 5 — Figure 5: Comparison of baseline segmentation networks (U-Net and UNETR) and their PCCA-integrated counterparts, PROMISE-C NN and PROMISE-T xformer , across cross-modal and cross-anatomical datasets. In (a)–(c), ground-truth boundaries are shown in green and predictions in red for binary segmentation tasks. In (d), for multi-class cardiac segmentation on CAMUS, green contours denote ground truth, while blue, purple, and yellow indicate predicted MYO, LV, and LA boundaries, respectively. PROMISE-Net variants exhibit improved anatomical fidelity, smoother contours, and reduced fragmentation compared to their baselines.

Model	DSC	FLOPs	Params
U-Net	$82.2$	$96.6$	$31$
SAM	$85.1$	$2991.32$	$312$
PROMISE-C NN	$\boldsymbol{90.7}$	$\boldsymbol{118.15}$	$\boldsymbol{61}$

Table 5: Computational efficiency, model complexity, and performance comparison of selected segmentation architectures. DSC is reported on ISIC-2017; bold blue denotes the best result per metric.

Panel of Figure 6 — Figure 6: Bland–Altman plots comparing predicted and reference EDV, ESV, and EF for UNETR (a-c) and our PROMISE-T xformer (d-f). The proposed model reduced the mean bias (EDV: $-13.9\rightarrow-6.7$ mL; EF: $-4.15\rightarrow-2.08$ %) and narrowed the LoA across all indices, demonstrating improved volumetric precision and enhanced temporal coherence. The EF agreement ( $-18.8$ % to $+14.6$ %) lies within the inter-observer variability range, confirming clinically reliable ventricular function estimation.

Figure 7: Comparison of segmentation performance between UNETR and PROMISE-T xformer on CAMUS: class-wise DSC (LV, MYO, and LA) and DSC across good, medium, and poor image quality levels Leclerc et al. [2019] .

Figure 8: Dice score comparison using observer-drawn and automatically generated bounding-box prompts, with pairwise effect sizes (Cohen’s $d$ ).

Figure 9: Qualitative analysis of challenging and failure cases on ISIC-2017 and Kvasir-Polyp. Each column compares U-Net, PROMISE-C NN , UNETR, PROMISE-T xformer , and SAM. Low-contrast textures, disconnected regions, and complex boundary structures lead to degraded performance across all methods; however, PROMISENet variants consistently preserve better boundary conformity than baseline U-Net and UNETR under extreme cases.

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 788 papers

Jul 8, 2026

HPR-SAM: Hierarchical Probabilistic Representation Learning for Prompt-free SAM-based Medical Image Segmentation

Prompt-free adaptation of the Segment Anything Model (SAM) has emerged as a promising paradigm for automatic medical image segmentation. Existing methods mainly focus on prompt generation, while overlooking that prompt quality is fundamentally constrained by the expressiveness of anatomical representations. However, deterministic prototypes or semantic tokens are insufficient to jointly capture global anatomical priors, intra-structure diversity, and local structural reliability. To address this limitation, we propose the Hierarchical Probabilistic Representation (HPR) framework, which learns complementary anatomical representations through Distributional Anatomical Representation (DAR), Multi-component Anatomical Representation (MAR), and Local Reliability Representation (LRR), and integrates their predictions via Hierarchical Prediction Fusion (HPF) while remaining compatible with the original SAM decoder. Experiments on the Synapse, LA, and PROMISE12 datasets demonstrate that HPR-SAM achieves state-of-the-art performance on Synapse and the best performance under few-shot settings on LA and PROMISE12, validating the effectiveness of the proposed hierarchical probabilistic representation learning framework for prompt-free medical image segmentation. Code is available at https://anonymous.4open.science/r/HPR-SAM-E4AF.

Yingzhen Hu, Yiheng Zhong, Keying Zhu +5

Yale University, United States

Medical Imaging Foundation Models Representation Learning Hierarchical Representation Learning Medical Image Segmentation

Ask PDF

Sep 21, 2025

SAM-DCE: Decoupled Context Enhancement for Prompt-Free Medical Image Segmentation

The Segment Anything Model (SAM) is a promising foundation model for medical image segmentation, but its prompt-driven design is not directly suitable for fully automatic multi-class anatomical segmentation. Without manual point, box, or mask prompts, SAM faces a dual over-smoothing problem: deep image features may lose fine local structural evidence, while mask tokens may become semantically ambiguous. We propose SAM-DCE, a decoupled context enhancement framework for prompt-free medical image segmentation. SAM-DCE introduces Spatial Representation Compensation (SRC) to recover anatomical evidence from intermediate ViT features and Semantic Token Calibration (STC) to improve the category consistency of foreground mask tokens. Rather than replacing the pretrained decoder or fusing output masks, SAM-DCE refines the spatial and semantic representations before recombining them through SAM's original token-to-mask mechanism. On the Synapse multi-organ CT segmentation dataset, SAM-DCE achieved a mean Dice score of 83.86% and an HD95 of 16.04 mm, outperforming the representative CNN-, Transformer-, and SAM-based baselines evaluated here. Ablation experiments further showed that SRC and STC provide complementary improvements, supporting the proposed spatial--semantic factorization for prompt-free SAM adaptation.

Yingzhen Hu, Yiheng Zhong, Aijing Yang +2

Mohamed bin Zayed University of AI, Abu Dhabi, UAE · Xi’an Jiaotong-Liverpool University, Suzhou, China · University of Liverpool, Liverpool, United Kingdom

Medical Image Segmentation Segment Anything Models Multi-Organ Segmentation

Ask PDF

Apr 14, 2026

Scaling In-Context Segmentation with Hierarchical Supervision

In-context learning (ICL) enables medical image segmentation models to adapt to new anatomical structures from limited examples, reducing the clinical annotation burden. However, standard ICL methods typically rely on dense, global cross-attention, which scales poorly with image resolution. While recent approaches have introduced localized attention mechanisms, they often lack explicit supervision on the selection process, leading to redundant computation in non-informative regions. We propose PatchICL, a hierarchical framework that combines selective image patching with multi-level supervision. Our approach learns to actively identify and attend only to the most informative anatomical regions. Compared to UniverSeg, a strong global-attention baseline, PatchICL achieves competitive in-domain CT segmentation accuracy while reducing compute by 44% at

512\times512

resolution. On 35 out-of-domain datasets spanning diverse imaging modalities, PatchICL outperforms the baseline on 6 of 13 modality categories, with particular strength on modalities dominated by localized pathology such as OCT and dermoscopy. Training and evaluation code are available at https://github.com/tidiane-camaret/ic_segmentation

T. Camaret Ndir, Marco Reisert, Robin T. Schirrmeister

Division of Medical Physics, Department of Radiology, Medical Center – University of Freiburg, Freiburg, Germany

Medical Image Segmentation Cross-Modality Medical Image Segmentation

Ask PDF

Abstract

Medical Image Segmentation

Explore similar work

HPR-SAM: Hierarchical Probabilistic Representation Learning for Prompt-free SAM-based Medical Image Segmentation

SAM-DCE: Decoupled Context Enhancement for Prompt-Free Medical Image Segmentation

Scaling In-Context Segmentation with Hierarchical Supervision