DeCoDrift: Stabilizing Decoder Coupling in Closed-Loop Foundation Segmentation

Segmentation is central to clinical diagnosis and monitoring, yet the reliability of modern foundation models in medical imaging still depends on the availability of precise prompts. The Segment Anything Model (SAM) offers powerful zero-shot capabilities, although it collapses under the weak, generic, and noisy prompts that dominate real clinical workflows. In practice, annotations such as centerline points are coarse and ambiguous, often drifting across neighboring anatomy and misguiding SAM toward inconsistent or incomplete masks. We introduce SPD, a Saliency-Guided Prompt Distillation framework that converts these unreliable cues into robust guidance. SPD first learns data-driven anatomical priors through a lightweight saliency head to obtain confident localization maps. These priors then drive Contextual Prompt Distillation, which validates and enriches noisy prompts using cues from anatomically adjacent slices, producing a consensus prompt set that matches the behavior of expert reasoning. A Pairwise Slice Consistency objective further enforces local anatomical coherence during segmentation. Experiments on four challenging MRI and CT benchmarks demonstrate that SPD consistently outperforms existing SAM adaptations and supervised baselines, delivering large gains in both region-based and boundary-based metrics. SPD provides a practical and principled path toward reliable foundation model deployment in clinical environments where only imperfect prompts are available.

Promptable segmentation foundation models such as SAM3 accept an open-vocabulary text concept and return every instance matching it, but adapting them to a specialized domain by full fine-tuning is computationally prohibitive for the organizations that would benefit most. This study applies Low-Rank Adaptation (LoRA) to SAM3 for multi-class structural defect segmentation and examines both how such a model can be supervised from conventional annotation and whether the resulting efficiency gain transfers across datasets. Two contributions are methodological. First, we describe a supervision procedure that trains a concept-promptable model directly from COCO-style class-labeled instance segmentation by using the category name itself as the prompt, requiring no prompt templates, no synonym expansion, and no learned class embeddings. Second, we identify and mitigate a failure mode specific to this setting: because a conventional annotation file yields positive prompts exclusively, the model's presence prediction decouples from the text condition and degenerates into responding to any prompt, a collapse that is invisible to every metric computed on positive prompts alone. Exhaustive hard-negative prompting, in which every dataset category absent from an image is issued as a zero-detection query, addresses this at no annotation cost. Two adapter placements were compared under an identical protocol, updating 0.121% and 1.341% of model parameters. On a purpose-built tunnel lining dataset, pixel intersection-over-union improved from 0.017 to 0.338 and instance-level recall from 0.375 to 0.672; on the independent public Structural Defects Dataset, from 0.017 to 0.855 and from 0.574 to 1.000. Improvements were directionally consistent across ten metrics on both datasets, and the largest per-category gains occurred precisely where zero-shot competence was absent.

High-resolution 3D segmentation of hip and shoulder anatomy from CT and MRI is essential for surgical planning, yet frozen segmentation models often fail under domain shift. CNN-based expert models are fully automatic but lack adaptability, whereas promptable foundation models generalize better but require manual prompting. We present MedSAM2-Anatomy, a training-free inference-time optimization framework that improves frozen segmentation models without retraining or human interaction. A frozen expert model generates anatomical priors that are automatically converted into multiple prompt hypotheses for a frozen 3D foundation model. Candidate masks are fused while anatomically implausible priors are rejected. No model weights are updated and no manual prompts are required. TotalSegmentator and MedSAM2 are used as representative expert and foundation models, allowing the contribution of the inference policy to be isolated. Evaluation on the independent Balgrist-V0 CT and MRI cohorts shows that inference-time optimization increases median Dice from 0.71 to 0.92 on hip MRI and from 0.89 to 0.92 on shoulder CT, while reducing median HD95 on hip MRI from 22.0 mm to 5.0 mm. On public TotalSegmentator benchmarks, the expert model remains strongest, indicating that the optimal fusion strategy depends on the reliability of the expert prior. These results demonstrate that training-free inference-time optimization provides a practical strategy for improving frozen segmentation models without manual prompting.