GET: Generative Embedding Translation for Medical Image Segmentation

We present LlamaSeg, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions. By reformulating segmentation as visual generation, LlamaSeg encodes masks as visual tokens and uses a LLaMA-style Transformer for direct next-token prediction, naturally fitting segmentation into autoregressive architectures. To support large-scale training, we introduce a data annotation pipeline and construct the SA-OVRS dataset, which contains 2M segmentation masks annotated with over 5,800 open vocabulary labels or diverse textual descriptions, spanning diverse real-world scenarios. This enables our model to localize objects in images based on text prompts and to generate fine-grained masks. We further introduce the composite metric average Hausdorff Distance ($d_{\mathrm{AHD}}$) to evaluate mask contour fidelity for generative models better. Experiments show that LlamaSeg consistently outperforms existing generative approaches on multiple segmentation benchmarks and delivers finer, more accurate segmentation masks. Code and dataset are available at https://github.com/GML-FMGroup/llamaseg.

Recent approaches for segmentation have leveraged pretrained generative models as feature extractors, treating segmentation as a downstream adaptation task via indirect feature retrieval. This implicit use suffers from a fundamental misalignment in representation. It also depends heavily on indirect feature extraction pipelines, which complicate the workflow and limit adaptation. In this paper, we argue that instead of indirect adaptation, segmentation tasks should be trained directly in a generative manner. We identify a key obstacle to this unified formulation: VAE latents of binary masks are sharply distributed, noise robust, and linearly separable, distinct from natural image latents. To bridge this gap, we introduce timesteps sampling strategy for binary masks that emphasizes extreme noise levels for segmentation and moderate noise for image generation, enabling harmonious joint training. We present GenMask, a DiT trains to generate black-and-white segmentation masks as well as colorful images in RGB space under the original generative objective. GenMask preserves the original DiT architecture while removing the need of feature extraction pipelines tailored for segmentation tasks. Empirically, GenMask attains state-of-the-art performance on referring and reasoning segmentation benchmarks and ablations quantify the contribution of each component.

Strongly paired medical image translation contains a substantial component that can be predicted directly from the source. The current reality is that pure deterministic prediction can smooth away fine detail, while generative models can recover detail but may also introduce unnecessary or potentially harmful variation. We propose PredRA, a fast framework that uses the deterministic prediction as a stable reference and further extracts additional deterministic components from the residual during generative refinement, thereby improving fidelity while maintaining perceptual quality. The goal is simple: we view the entire residual-based generative process as an optimization problem and derive a practical solution that recovers useful residual detail through controlled refinement while keeping the prediction close to the paired target. Mechanism studies further show that useful residual information follows structured patterns, but its usefulness is difficult to estimate reliably at the voxel level. We therefore globally control how much of the residual refinement is added to the deterministic prediction, thereby reducing the accumulation of unnecessary uncertainty. PredRA therefore combines deterministic component extraction with controlled stochastic refinement for fast, fidelity-preserving, and perceptually strong medical image translation. We validate the approach across multiple real-world datasets, showing that controlled refinement improves fidelity to the paired target compared with full residual refinement while retaining much of the perceptual benefit of generative modeling. PredRA achieves competitive or superior performance to substantially larger state-of-the-art models with 1.4-11.9x fewer total parameters and 3.1-26.2x fewer trainable parameters, while its 32-step flow sampler requires 31.25x fewer sampling steps than the matched 1000-step DDPM.