LUMOS: Latent Universal Medical Priors for Segmentation

Zhuonan Liang; Wei Guo; Jie Gan; Yaxuan Song; Runnan Chen; Hang Chang; Weidong Cai

cs.CVMar 1, 2026

LUMOS: Latent Universal Medical Priors for Segmentation

Authors: Zhuonan Liang, Wei Guo, Jie Gan, Yaxuan Song, Runnan Chen, Hang Chang, Weidong Cai

Organizations: The University of Sydney, Sydney, NSW 2006, Australia. · Biological System & Engineering Division, Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA. · Berkeley Biomedical Data Science Center, Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA.

Abstract

General vision foundation models (VFMs) have been primarily developed on natural images, and their utility for medical image segmentation is therefore often considered to depend on costly adaptation or domain-specific fine-tuning. In this paper, we revisit this assumption from a different perspective: rather than requiring VFM segmentors to relearn visual regularities, we investigate whether the low-level visual priors necessary for anatomical delineation already lie dormant within general VFMs. We observe that frozen VFMs, despite lacking medical supervision, encode transferable visual regularities. These properties are not exclusive to natural images but are also fundamental to medical image understanding. Motivated by this observation, we propose Latent Universal Medical PriOrs for Segmentation (LUMOS), a novel framework that amplifies general VFM priors to conventional medical segmentors. LUMOS consists of two key components: (1) Pathfinder that distills visual cues from a frozen vision foundation model, and (2) Inspiror that sparks the conventional medical networks with spatial guidance from distilled visual regularities. In this way, the segmentor is relieved from learning complex visual regularities entirely from limited medical annotations and can instead focus on task-specific anatomical delineation. Across diverse medical datasets and token-based VFMs, LUMOS shows that general VFMs can serve as spatial prior generators when their frozen token spaces preserve patch-level pattern relevance. DINO provides stable matched-backbone gains, while SigLIP exposes VFM-specific sensitivity caused by its different token granularity and representation objective.

Figures & tables

Fig. 1: Motivation of the proposed LUMOS framework. We observe that frozen vision foundation models, despite lacking medical supervision, encode transferable visual regularities that are fundamental to medical image understanding. By leveraging these priors as spatial guidance, we can enhance the performance of conventional medical segmentation networks without requiring extensive fine-tuning of the foundation model.

Fig. 2: Overview of the proposed LUMOS framework. A frozen token-based VFM, instantiated by DINOv3 in the main experiments, serves as a guide generator by extracting dense token features that are converted into spatial guide masks via the Reg Book mechanism. These guide masks are then used to gate feature activations in the segmentation backbone, which produces the final segmentation output.

Method	Venue	Kvasir-SEG			ISIC 2017			TN3K
Method	Venue	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$
nnUNet [ 12 ]	Nat. Methods’21	83.92	89.71	20.85	82.30	89.18	13.38	72.81	82.38	33.47
SwinUNet [ 32 ]	ECCVW’22	40.75	54.42	54.42	81.24	88.38	14.17	52.28	64.89	26.72
H2Former [ 33 ]	TMI’23	83.87	89.50	19.81	8 2.56	89.36	13.04	73.94	82.67	2 1.91
U-KAN [ 9 ]	AAAI’25	63.81	72.17	36.92	74.62	83.41	23.57	69.60	78.66	24.49
nnWNet [ 11 ]	CVPR’25	8 4.58	8 9.98	1 8.17	82.62	8 9.44	12.73	66.46	76.02	38.61

TABLE I: Baseline comparison across venues. LUMOS-W (DINO) achieves the best Dice on all datasets and the best IoU on Kvasir-SEG and TN3K.

Backbone	Method	Kvasir-SEG			ISIC 2017			TN3K
Backbone	Method	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$
UNet	Baseline [ 10 ]	70.29	79.16	41.58	72.95	81.87	41.58	70.65	79.45	24.59
	LUMOS (SigLIP)	78.69 (+8.40)	86.79 (+7.63)	26.33 (-15.25)	76.82 (+3.87)	86.71 (+4.84)	16.59 (-24.99)	69.63 (-1.02)	79.50 (+0.05)	28.13 (+3.54)
	LUMOS (DINO)	83.18 (+12.89)	89.51 (+10.35)	17.78 (-23.80)	81.24 (+8.29)	88.78 (+6.91)	17.86 (-23.72)	72.09 (+1.44)	81.41 (+1.96)	24.97 (+0.38)
nnWNet	Baseline [ 11 ]	84.58	89.98	18.17	82.62	89.44	12.73	66.46	76.02	38.61
	LUMOS (SigLIP)	82.35 (-2.23)	89.31 (-0.67)	21.72 (+3.55)	81.01 (-1.61)	88.76 (-0.68)	14.40 (+1.67)	72.52 (+6.06)	82.57 (+6.55)	24.30 (-14.31)

TABLE II: Matched-backbone ablation. Deltas relative to each UNet or nnWNet baseline show that DINO-based LUMOS consistently improves Dice, with larger gains in weaker settings.

Method	Kvasir-SEG			ISIC 2017			TN3K
Method	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$
Seg-DINO [ 5 ]	80.64	89.65	18.62	77.60	85.76	20.80	74.43	83.18	18.62
Seg-LUMOS	79.10 (-1.54)	87.23 (-2.42)	18.64 (+0.02)	81.08 (+3.48)	89.23 (+3.47)	16.42 (-4.38)	61.67 (-12.76)	73.08 (-10.10)	16.13 (-2.51)
Seg-LUMOS (SPE)	87.37 (+6.73)	92.72 (+3.07)	12.16 (-6.46)	79.54 (+1.94)	88.11 (+2.35)	16.56 (-4.24)	74.58 (+0.15)	83.56 (+0.38)	21.41 (+2.79)
nnWNet [ 11 ]	84.58	89.98	18.17	82.62	89.44	12.73	66.46	76.02	38.61
LUMOS-W	84.82 (+0.24)	90.86 (+0.88)	17.50 (-0.67)	82.32 (-0.30)	89.81 (+0.37)	13.14 (+0.41)	74.48 (+8.03)	83.40 (+7.38)	24.08 (-14.53)

TABLE III: Optional VFM-specific pattern enhancement for DINO-derived guidance. The enhancement improves several metrics but remains dataset-dependent; matched frozen-VFM LUMOS-W remains the main guide ablation.

Fig. 3: Qualitative results for segmentation. First row : original input image. Second row : actual segmentation masks. Third row : guidance mask from LUMOS. The guidance mask highlights the rough location of the target, which can assist the segmentation backbone in focusing on relevant regions.

Method	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$
Baseline-W	84.58	89.98	18.17
Direct fused	83.82 (-0.76)	88.44 (-1.54)	22.01 (+3.84)
Soft gating	84.82 (+0.24)	91.02 (+1.04)	17.50 (-0.67)

TABLE IV: Guide-injection design on Kvasir-SEG. Soft gating improves Dice and HD95 over direct fusion and the nnWNet-based baseline.

Guide Mask Size	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$
$8\times 8$	73.34	82.69	24.65
$11\times 11$	72.80	81.94	24.76
$22\times 22$	74.48	83.40	24.08
$32\times 32$	72.81	82.19	25.15

TABLE V: Guide-mask resolution on TN3K in the compact main-text sweep. The

22\times 22

mask gives the best final IoU and DSC among these reported settings and is selected as the overlap-oriented tradeoff.

	Kvasir-SEG			ISIC 2017			TN3K
Method	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$	IoU $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$
nnWNet	72.56	81.59	37.34	72.20	80.93	41.02	66.41	75.33	56.27
LUMOS-W (DINO)	80.50 (+7.94)	87.34 (+5.75)	19.91 (-17.43)	75.09 (+2.89)	85.02 (+4.09)	28.92 (-12.10)	67.43 (+1.02)	78.20 (+2.87)	33.67 (-22.60)

TABLE VI: Reduced-annotation setting with 30% training masks. LUMOS-W improves nnWNet across datasets, with especially large Kvasir-SEG gains and lower HD95 on all three benchmarks.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 4: Low-annotation learning curves with 30% training masks. Columns show Kvasir-SEG, ISIC 2017, and TN3K; rows show IoU, DSC, and HD95. LUMOS-W converges faster than nnWNet and maintains stronger boundary-distance behavior under limited dense supervision.

$s_{G}$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
4	82.30	73.10	23.98
8	82.70	73.30	24.65
11	81.90	72.80	24.76
16	83.10	74.00	24.22
22	83.10	73.70	24.08
32	82.20	72.80	25.15

Appendix

TABLE VII: TN3K guide-mask side-length ablation.

s_{G}

denotes the guide-mask side length.

$s_{G}$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
8	90.10	83.90	35.52
16	90.40	84.20	38.12
24	90.30	84.20	41.31
32	90.40	84.20	39.13
44	90.80	85.00	35.29
64	90.80	84.90	38.93

Appendix

TABLE VIII: Kvasir-SEG guide-mask side-length ablation.

s_{G}

denotes the guide-mask side length.

$K$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
64	81.90	72.70	25.41
128	82.50	73.40	24.86
256	82.50	73.30	24.44
512	82.50	73.30	23.86
1024	82.60	73.50	24.69
2048	82.70	73.40	25.07

Appendix

TABLE IX: TN3K Reg Book prototype-count ablation.

K

denotes the number of Reg Book prototypes.

$K$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
64	90.50	84.30	39.94
128	90.70	84.60	40.47
256	90.00	83.80	39.22

Appendix

TABLE X: Kvasir-SEG Reg Book prototype-count ablation with available completed local runs.

K

denotes the number of Reg Book prototypes.

Mode	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
Soft gate (ours)	90.20	84.00	40.29
Hard mask	0.00	0.00	830.60
Concatenate	91.00	84.90	38.31

Appendix

TABLE XI: Kvasir-SEG guidance-application mode ablation.

Method	Total (M)	Trainable (M)
UNet	7.76	7.76
nnWNet	7.04	7.04
nnUNet-style	71.82	71.82
SwinUNet	27.17	27.17
SegDINO	24.07	2.48
LUMOS-U-DINO	29.90	8.30

Appendix

TABLE XII: Parameter counts for implemented and external baseline methods. For frozen-encoder models, trainable parameters reflect the default fine-tuning policy used in our runs.

Fig. 5: TN3K held-out prediction comparison from local checkpoints. UNet, nnWNet, and LUMOS were regenerated from local checkpoints; LUMOS uses the completed guide-size run with $s_{G}=64$ .

Fig. 6: TN3K training visualizations for guide-mask side-length settings $s_{G}$ .

Fig. 7: Kvasir-SEG training visualizations for guide-mask side-length settings $s_{G}$ .

Fig. 8: TN3K training visualizations for Reg Book prototype-count settings $K$ .

Fig. 9: Kvasir-SEG training visualizations for Reg Book prototype-count settings $K$ with available completed local runs.

Fig. 10: Kvasir-SEG training visualizations for guidance-application modes.

Explore similar work

Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models
Aug 2, 2026Myeongkyun Kang, Yanting Yang, Xiaoxiao LiMedical Imaging Foundation Models Medical Image Grounding
The University of British Columbia · Vector Institute
RAP: Retrieve, Adapt, and Prompt-Fit for Training-Free Few-Shot Medical Image Segmentation
Mar 29, 2026Zhihao Mao, Bangpu ChenMedical Image Segmentation Few-Shot Medical Image Segmentation
School of Computer Science, China University of Geosciences, Wuhan, China
MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation
Aug 13, 2026Rafi Ibn Sultan, Hui Zhu, Chengyin Li +1Medical Image Segmentation Text-Guided Image Segmentation
Department of Computer Science Wayne State University · Department of Radiation Oncology Henry Ford Health · Institute for AI and Data Science Wayne State University

Aug 2, 2026J/K move · Enter open · S save

Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models

Myeongkyun Kang, Yanting Yang, Xiaoxiao Li

The University of British Columbia · Vector Institute

Medical Imaging Foundation Models Medical Image Grounding Visual Grounding Medical Image Representation Learning Fine-Grained Visual Perception

Open paper AskPDF arXiv

Fine-grained visual representations are essential for medical image analysis, particularly when diagnostically relevant evidence is subtle and spatially localized. Modern transformer-based medical vision encoders must therefore learn patch-level representations that are both clinically meaningful and spatially consistent. Without these properties, large vision-language models (LVLMs) operate on an ambiguous visual foundation, limiting their ability to generate clinically reliable and spatially grounded responses. However, existing training strategies for medical vision encoders rarely achieve both objectives. Image-text alignment provides clinically meaningful supervision primarily at the image level, leaving the spatial localization of diagnostic evidence weakly constrained. In contrast, self-supervised learning promotes spatial consistency but lacks the semantic supervision needed to distinguish visually similar yet clinically distinct regions. To address this gap, we present LoFi, a medical vision foundation model built on location-aware fine-grained representation learning. LoFi trains a vision encoder with a lightweight large language model under grounding and grounded captioning objectives. Because these objectives require predicting location from clinical text and vice versa, spatial consistency emerges without any explicit patch-level regularization. To enable training at scale, we construct MedG, a large-scale medical grounding dataset of 4.48M image-text-box triplets curated from 84 datasets spanning 7 modalities. Across phrase grounding, visual question answering, and region-based organ classification under perturbations, LoFi consistently outperforms general-purpose and medical vision foundation models as well as state-of-the-art LVLMs. Code is available at https://github.com/myeongkyunkang/lofi-medg.

Abstract

Explore similar work

Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models

RAP: Retrieve, Adapt, and Prompt-Fit for Training-Free Few-Shot Medical Image Segmentation

MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation