GET: Generative Embedding Translation for Medical Image Segmentation

Md Maklachur Rahman; Md Hasan Al Banna; Saraf Anjum; Mahmudul Hasan; Tracy Hammond

eess.IVAug 23, 2026

GET: Generative Embedding Translation for Medical Image Segmentation

Authors: Md Maklachur Rahman, Md Hasan Al Banna, Saraf Anjum, Mahmudul Hasan, Tracy Hammond

Organizations: Texas A&M University, College Station, TX, USA · Independent Researcher · Alfa Laval

Abstract

Generative segmentation provides an alternative to direct pixel-wise prediction by operating on learned latent representations, but effective image-to-mask translation must preserve target structure while remaining computationally efficient. We propose Generative Embedding Translation (GET), a structured embedding-translation framework that progressively transforms image embeddings into mask embeddings within the frozen latent space of a Stable Diffusion VAE. GET uses a U-Net-style Embedding Translation Network with 1.07M trainable parameters, combining Mobile Bottleneck Convolutions, Subsampled Self-Attention, and Multi-scale Feature Enrichment for local modeling, global context, and multi-scale refinement. Across five medical segmentation datasets, GET outperforms generative, CNN, and Transformer baselines. Compared with the strongest generative baseline, GMS, GET improves average Dice and IoU by 0.93% and 1.26%, reduces HD95 by 0.81 pixels, and uses 31.41% fewer trainable parameters. Under bidirectional BUS-BUSI domain shift, GET further improves Dice and IoU by 3.51% and 3.39%, while reducing HD95 by 27.37 pixels. Our code is available at: https://github.com/maklachur/GET.

Figures & tables

Figure 1: Overview of the proposed framework. (a) Main pipeline: a frozen SD-VAE encodes both image ( $I$ ) and mask ( $M$ ) into latent features $\mu_{\text{img}}$ and $\mu_{\text{mask}}$ . The lightweight ETN translates image latent $\mu_{\text{img}}$ to segmentation latent $\hat{\mu}_{\text{seg}}$ . The $s_{1}$ , $s_{2}$ , $s_{3}$ , and $out$ are the segmentation latents from different stages of ETN. The $\hat{\mu}_{\mathrm{seg}}^{(s_{1})}$ and $\hat{\mu}_{\mathrm{seg}}^{(out)}$ denote the segmentation latent for $s_{1}$ and $out$ , respectively, which are then decoded to generate the final masks. (b) ETN integrates MBC, MFE, and SSA modules for latent translation. (c) MBC acts as the encoder within ETN. (d) MFE aggregates multi-scale features via parallel dilated branches in the decoder. (e) SSA operates at the bottleneck, capturing long-range context through subsampled attention.

AT	Model	Trainable Params (M) $\downarrow$	BUS			BUSI
AT	Model	Trainable Params (M) $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
C	UNet [ 28 ]	14.03	81.65 .50	70.98 .73	17.50 1.90	72.42 .79	63.18 1.05	35.22 4.10
C	MultiResUNet [ 15 ]	7.26	80.56 .47	70.54 .69	19.04 2.02	72.58 .82	62.77 1.08	33.99 3.85
C	ACC-UNet [ 14 ]	16.77	83.30 .51	73.36 .77	16.62 1.73	77.34 .65	68.71 .92	25.26 2.85
C	nnUNet [ 16 ]	20.6	85.61 .36	78.53 .58	11.56 1.15	79.60 .59	71.20 .86	21.89 2.41
C	EGE-UNet $\star$ [ 29 ]	0.05	72.94 .75	62.15 .98	27.57 3.05	75.32 .89	60.42 1.14	29.30 3.42

Table 1: Quantitative results on BUS and BUSI datasets. AT denotes the primary architecture type: C = CNN, T = Transformer, G = Generative.

\uparrow

indicates higher is better,

\downarrow

lower is better, and

\star

marks models with fewer parameters than ours. All SOTA baselines are reproduced using their public code and papers, and trained and tested on the same train–test splits as our model. Results are averaged over five independent runs and reported as mean

\pm

SD. Best scores are shown in bold.

Model	GlaS			Kvasir–Instrument			HAM10000
Model	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
UNet	88.08 .47	80.17 .72	18.27 1.96	93.86 .29	89.30 .49	8.69 .79	92.36 .29	87.14 .50	13.51 1.22
MultiResUNet	88.41 .50	80.45 .77	17.29 1.83	92.33 .27	87.06 .46	9.48 .86	92.64 .26	87.43 .46	13.29 1.17
ACC-UNet	88.72 .46	81.03 .70	16.92 1.69	93.96 .31	89.82 .57	8.71 .76	93.33 .25	88.67 .45	10.58 .92
nnUNet	87.30 .43	78.32 .68	19.98 2.12	93.97 .26	90.24 .45	8.50 .72	93.89 .28	89.43 .51	9.31 .79
EGE-UNet $\star$	83.20 .67	71.24 .99	28.87 2.98	92.62 .38	86.25 .68	9.06 .87	93.03 .31	87.97 .54	11.14 .98

Table 2: Results on GlaS, Kvasir–Instrument, and HAM10000 datasets. All SOTA baselines are reproduced using their public code and trained/tested on the same splits as ours. Results are averaged over five runs and reported as mean

\pm

SD.

Model	BUSI $\to$ BUS			BUS $\to$ BUSI
Model	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
UNet	63.20	51.11	46.50	53.71	47.81	98.07
MultiResUNet	61.61	50.25	53.71	56.24	50.32	92.40
ACC - UNet	64.71	52.39	42.08	47.61	46.68	136.11
nnUNet	78.33	67.84	20.90	59.18	53.75	88.27
EGE - UNet $\star$	68.90	56.81	33.94	54.37	50.39	106.08

Table 3: Cross-domain generalization results between BUS and BUSI. BUSI

\rightarrow

BUS denotes training on BUSI and testing on BUS, while BUS

\rightarrow

BUSI denotes training on BUS and testing on BUSI. Best results are highlighted in bold.

Figure 2: Qualitative comparison on BUS, BUSI, HAM10000, GlaS, and Kvasir–Instrument. We show the input image, ground-truth mask, and predictions from representative segmentation methods across the five datasets.

Modules			BUS			BUSI			GlaS
MBC	MFE	SSA	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
Baseline			88.48	80.05	6.72	79.23	70.73	21.06	87.38	78.56	18.82
✓			89.34	81.26	6.53	80.27	71.33	19.43	87.68	79.24	17.84
	✓		89.15	80.95	6.76	81.01	72.07	21.15	88.22	79.91	16.75
		✓	89.27	81.04	7.11	80.78	71.95	20.09	87.96	79.37	18.54
✓	✓		90.01	82.42	5.71	81.41	72.17	19.26	88.61	80.44	18.30

Table 4: Ablation of MBC, MFE, and SSA on BUS, BUSI, and GlaS. ✓ denotes module inclusion. Baseline is a U-Net–like ETN without MBC/MFE/SSA, using standard convolutional blocks. Best results are in bold.

Dilation Rates ( $d$ )	BUS			GlaS
Dilation Rates ( $d$ )	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
Two-branch
$d=\{1,2\}$	89.20	81.26	6.21	88.21	78.89	18.83
$d=\{2,4\}$	89.58	81.60	5.97	88.57	79.50	18.57
$d=\{1,6\}$	89.82	81.52	5.38	88.80	79.83	18.00
$d=\{6,12\}$	90.07	81.87	5.68	88.82	80.02	17.50

Table 5: Effect of dilation rate configurations (

d

) in the MFE module across BUS and GlaS datasets. Results are averaged over five runs.

Loss				BUS			GlaS
$\mathcal{L}_{\mathrm{dice}}$	$\mathcal{L}_{\mathrm{ftv}}$	$\mathcal{L}_{\mathrm{rec}}$	$\mathcal{L}_{\mathrm{la}}$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
✓	✓			90.09	82.29	8.62	87.05	78.40	20.43
✓	✓	✓		90.48	83.09	6.40	88.83	80.92	17.78
✓	✓		✓	90.38	82.81	8.07	88.05	79.96	17.81
		✓	✓	89.60	81.64	6.34	87.79	79.46	18.24
✓		✓	✓	90.18	82.58	5.30	88.70	80.87	17.11

Table 6: Ablation study of losses on BUS and GlaS.

\checkmark

indicates the inclusion of the loss term. Bold highlights the best results.

Aux Decoder(s)	BUS			GlaS
Aux Decoder(s)	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
None	88.51	79.93	6.99	89.07	81.35	17.14
s1	90.76	83.48	4.84	89.24	81.67	15.62
s2	90.28	82.60	5.26	89.18	81.45	17.41
s3	89.79	82.01	5.61	88.93	80.96	17.14
s1+s2	90.55	83.11	5.11	89.27	81.63	16.43

Table 7: Effect of auxiliary decoder heads (

s_{1},s_{2},s_{3}

) with the main output decoder head

out

at different stages on BUS and GlaS datasets. Best results are in bold.

Dataset	Dice $\uparrow$		IoU $\uparrow$		HD95 $\downarrow$
Dataset	SD	VQ	SD	VQ	SD	VQ
BUSI [ 2 ]	83.03	81.16	74.56	72.66	16.99	21.30
HAM10000 [ 32 ]	94.23	93.09	89.95	88.19	9.06	12.48
Kvasir-Instr. [ 17 ]	94.55	92.97	90.63	89.11	8.45	10.85

Table 8: Performance comparison on three datasets using different pre-trained VAE models. SD = SD-VAE and VQ = VQ-VAE.

Model	GFLOPs $\downarrow$	Throughput (samples/s) $\uparrow$	Speedup (×) $\uparrow$
MedSegDiff-V2 [ 39 ]	972.66	0.62	$1.0\times$
SDSeg [ 19 ]	655.34	15.04	$24.3\times$
GSS [ 5 ]	476.28	16.24	$26.2\times$
GMS [ 12 ]	343.85	27.68	$44.7\times$
GET (Ours)	341.14	28.07	$45.3\times$

Table 9: Inference complexity of generative segmentation models at

224\times 224\times 3

(batch=1) on an RTX 3090 Ti. GFLOPs are computed using THOP (MACs counted as FLOPs). Speedup (

\times

) denotes throughput relative to MedSegDiff-V2. Lower GFLOPs and higher throughput indicate better efficiency.

Figure 3: Comparison of segmentation performance versus trainable parameter count across SOTA methods, averaged over five medical imaging datasets: BUS, BUSI, GlaS, HAM10000, and Kvasir–Instrument. (a) Trainable parameters vs. average DSC score. (b) Trainable parameters vs. average IoU score. (c) Trainable parameters vs. average HD95 score. Higher DSC and IoU ( $\uparrow$ ) indicate better overlap accuracy, while lower HD95 ( $\downarrow$ ) reflects better boundary precision. Blue circles denote CNN/Transformer-based models, and orange squares denote generative models. GET (Ours) achieves a favorable efficiency–accuracy trade-off, with top-tier DSC, IoU, and HD95 performance using substantially fewer trainable parameters than existing SOTA methods.

$SSA(r)$	BUS			GlaS
$SSA(r)$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
1	90.40	82.89	5.19	89.03	81.27	16.42
2	90.76	83.48	4.84	89.24	81.67	15.62
4	88.72	80.53	6.45	87.63	79.44	18.18

Table 10: Effect of subsampling factor

r

in the SSA module on BUS and GlaS datasets. A moderate setting (

r=2

) yields the best trade-off between segmentation accuracy (DSC and IoU) and boundary precision (HD95). Results are averaged over five runs; best values are in bold.

\uparrow

indicates higher is better,

\downarrow

lower is better.

DS Weights ( $w_{k}$ )	BUS			GlaS
	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
Uniform weights: (1.0, 1.0, 1.0, 1.0)	86.42	78.72	11.62	86.37	77.37	21.82
Linear weights: (1.0, 0.75, 0.5, 0.25)	89.22	81.43	10.65	88.40	80.48	16.79
Monotonic weights: (1.0, 0.4, 0.3, 0.2)	90.76	83.48	4.84	89.24	81.67	15.62

Table 11: Effect of deep supervision weighting strategies (

w_{k}

) on auxiliary heads for

\mathcal{L}_{\mathrm{rec}}

loss across BUS and GlaS datasets. Results are averaged over five runs.

\uparrow

indicates higher is better,

\downarrow

lower is better. Best values are highlighted in bold.

$\lambda$	BUS			GlaS
$\lambda$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
0.1	89.83	81.55	7.15	87.86	79.51	17.26
0.3	90.12	82.41	5.94	88.15	80.00	17.49
0.4	90.40	82.89	5.19	88.63	80.57	18.50
0.5	90.76	83.48	4.84	89.24	81.67	15.62
0.6	90.28	82.73	5.76	89.06	81.12	16.04

Table 12: Effect of auxiliary head weight (

\lambda

) in

\mathcal{L}_{\mathrm{seg}}

loss across BUS and GlaS datasets. The weight

\lambda

controls the contribution of intermediate supervision from the auxiliary decoder head. Optimal performance is achieved at

\lambda=0.5

, balancing guidance strength and segmentation accuracy.

\uparrow

indicates higher is better,

\downarrow

lower is better. Results are averaged over five runs; best values are highlighted in bold.

$\lambda_{la}$	BUS			GlaS
$\lambda_{la}$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	IoU $\uparrow$	HD95 $\downarrow$
0.001	88.52	80.34	8.94	88.37	80.19	17.89
0.005	89.25	81.41	7.78	88.94	80.97	16.38
0.01	90.76	83.48	4.84	89.24	81.67	15.62
0.02	89.02	81.04	6.94	88.72	80.89	17.06
0.05	88.13	79.93	9.05	88.21	79.97	17.14

Table 13: Effect of the L2 latent alignment weight

\lambda_{\mathrm{la}}

\mathcal{L}_{\mathrm{la}}

across BUS and GlaS. The best performance is obtained at

\lambda_{\mathrm{la}}=0.01

. Results are averaged over five runs; best values are highlighted in bold.

\uparrow

indicates higher is better and

\downarrow

lower is better.

Image Segmentation

Image segmentation assigns pixels or voxels to objects and regions, supporting measurement, scene understanding, and decisions across image domains.

See all 399 papers

LlamaSeg: Image Segmentation via Autoregressive Mask Generation
May 26, 2025Jiru Deng, Tengjin Weng, Tianyu Yang +3Image Segmentation Text-Guided Image Segmentation
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University · Shenzhen University +2
GenMask: Adapting DiT for Segmentation via Direct Mask Generation
Mar 25, 2026Yuhuan Yang, Xianwei Zhuang, Yuxuan Cai +6Image Segmentation Vision Foundation Model Adaptation
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University · Alibaba Group · School of Artificial Intelligence, Shanghai Jiao Tong University +1
PredRA: Fast Medical Image Translation by Deterministic Component Extraction and Controlled Stochastic Refinement
Sep 25, 2026Jianhai Zhang, Pattarawut Charatpangoon, Donghao Zhang +4Image-to-Image Translation Medical Image Synthesis
Department of Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada. · Ron and Rene Ward Centre for Healthy Brain Aging Research, Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada. · Department of Biomedical Engineering, School of Life Science and Technology, Huazhong University of Science and Technology, Wuhan, Hubei, China. +1

May 26, 2025J/K move · Enter open · S save

LlamaSeg: Image Segmentation via Autoregressive Mask Generation

Jiru Deng, Tengjin Weng, Tianyu Yang, Wenhan Luo+2

Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University · Shenzhen University+2

Image Segmentation Text-Guided Image Segmentation Promptable Image Segmentation Open-Vocabulary Semantic Segmentation

Open paper AskPDF arXiv

We present LlamaSeg, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions. By reformulating segmentation as visual generation, LlamaSeg encodes masks as visual tokens and uses a LLaMA-style Transformer for direct next-token prediction, naturally fitting segmentation into autoregressive architectures. To support large-scale training, we introduce a data annotation pipeline and construct the SA-OVRS dataset, which contains 2M segmentation masks annotated with over 5,800 open vocabulary labels or diverse textual descriptions, spanning diverse real-world scenarios. This enables our model to localize objects in images based on text prompts and to generate fine-grained masks. We further introduce the composite metric average Hausdorff Distance (

d_{\mathrm{AHD}}

) to evaluate mask contour fidelity for generative models better. Experiments show that LlamaSeg consistently outperforms existing generative approaches on multiple segmentation benchmarks and delivers finer, more accurate segmentation masks. Code and dataset are available at https://github.com/GML-FMGroup/llamaseg.

Abstract

Image Segmentation

Explore similar work

LlamaSeg: Image Segmentation via Autoregressive Mask Generation

GenMask: Adapting DiT for Segmentation via Direct Mask Generation

PredRA: Fast Medical Image Translation by Deterministic Component Extraction and Controlled Stochastic Refinement